Weekly Updates for Arjun Dahal

Hello Everyone,

Action items completed:

After a while of tinkering, I am able to reduce the change in unintended sensitive attributes to a much smaller percentage. The number is even lower if I don’t consider the change within the unintended sensitive attribute ( for example, within unprivileged races ). However, in the Credit dataset, the change is still high.

Today, I will present the results: ATN, change in other sensitive attributes, and number of fairness violations. If the results are good enough, we can move forward with the current results.

great to hear. look forward to hearing what you did to improve the results during friday’s meeting. if i recall correctly, now you did three datasets? we need to have more datasets and also perhaps more models. in general, reviewers want to have diversity in datasets and models. this can also help verify the validity of our approach.

Sure, Dr. Lei. I will work on it.

Hello Everyone,

I followed up on the action items discussed last Friday - primarily understanding the adversarial training process, and verifying the implement.

I made few improvements on the existing implementation. However, the accuracy/RMSE of the classifier/regressor of the adversarial process is almost constant after first few epochs. I am training the adversarial objective with higher weightage to see if the results can be improved.

I will take 20 mins to present the results & the issues in today’s meeting.

it is normal to have a technical obstacle during implementation. try to think about what is the root cause, and then what you can do about it. also try to be systematic. one suggestion is that you may want to have some hypotheses and then design some small experiments to verify them, one at a time.

Hello Everyone,

Milestone: Apr 8 ( Resolve the issues in the current implementation & Complete with the necessary dataset )

I have been working on the current project for a while now.

Last week I presented the issues I was currently facing in the project related to flipping of multiple sensitive attributes when only one is conditioned on the fader network. I have been investigating the problem, however, I have yet to understand the issue completely.

Additionally, I have presented on the IWCT2025, the preparation for which took a reasonable amount of time.

what is this issue? is this the one you discussed with me on friday afternoon or something new?

The issue is the one I discussed on Friday Dr. Lei.

Hi Group,

This week, I have been working on the implementation of my current work on Fairness Testing. I have some initial results however, there are some issues I am facing. Once I have more results, I will update my Slack channel with them.

I have a new idea presentation tomorrow, where I will be presenting this work:

Chen, Canyu, et al. “When fairness meets privacy: Fair classification with semi-private sensitive attributes.” arXiv preprint arXiv:2207.08336 (2022).

Hello Everyone,

Milestone: Complete the current project and have a draft ( ~ Apr 2 )

I was traveling during the spring break.

After my return, I worked on the existing implementation. I have nearly completed the implementation for the current project.

I will be sharing the results of end-to-end implementation in the meeting or by the end of the day.

I will be updating the status of the progress every day at the end of the day till the completion of the project.

make a commitment by first removing ~ in the target date

Hello Everyone,

This week I created the artifacts for IWCT 2025. I also carried out the literature review for the paper I am currently working on.

Currently, I am still focused on implementation of the current project I am working on.

I am lagging behind in the project, I will update status of daily progress everyday till the completion of the first draft in my channel.

try to be iterative. try to have a baseline version first and then improve upon it.

Hello Everyone,

This week, I focused on the problem of changing the sensitive attribute of tabular datasets, for which I studied a couple of possible techniques ( along with their pros and cons ) and presented them. Furthermore, I worked on the comments from the previous paper IWCT and made the source code ready to work in a virtual environment along with the necessary document that instructs how to run the code. I have revisited the implementation of the Fader networks.

I am looking into the papers that I could compare our work with regarding counterfactual fairness.

sounds good. i just sent you some comments on the iwct paper from nist.

Hello Group,

Milestone: Complete the undergoing project ( ~ Mar 4 ).

Action Item Completed:

1. Looked into the literature for papers relevant to changing the sensitive attribute.

1. Implemented Fader Networks for tabular data.

I will present some results today regarding the flipping of the sensitive attribute and take action based on today’s discussion.

sounds good. i suggest you target ICSE which has the submission deadline in about one week

Hello Everyone,

This week, I have been reading some literature for my use case, flipping sensitive attributes for fairness testing respecting domain constraints.

I found this work to be closely related to my work:

Lample, Guillaume, et al. “Fader networks: Manipulating images by sliding attributes.” Advances in neural information processing systems 30 (2017). I am looking forward to a brief discussion tomorrow.

try to make your report more informative. what are the major findings, and what you consider this paper to be related.

Hello Everyone,

Milestone: Feb 7 ( Baseline ): Includes end-to-end implementation of two types of VAE models

Action Items completed from the past week:

Conditional VAE was used similarly to how a conditional VAE is used for image data, and the ATN found was ~0.75 for the adult income dataset.

The current implementation needs to be improved for end-to-end results.

I will present tomorrow for the New Idea session.

what do you mean by end-to-end implementations? does it mean end-to-end for fairness testing? in the following days, i suggest you focus on the topic of how to flip sensitive attributes. first try to find all the existing work on this problem or related to this problem. second try to develop an algorithm to do the flipping. ask the question, what type of fairness would your algorithm/end-to-end approach test? third try to have an initial implementation to validate the algorithm.

Hello Everyone,

Milestone: Feb 7 ( Baseline )

The end-to-end implementation is carried out to understand the project better and find the potential issues during each step.

Action Items completed this week:

1. I looked into the literature for different types of disentangled VAEs and have a high-level idea of the types of disentangled VAEs available.

1. I have carried out an experiment to see the results on the naturalness of the generated t-way samples using beta-VAE for different beta values.

• We obtain better naturalness till a certain value of beta, after which the naturalness decreases.

• I will share more findings in tomorrow’s meeting.

Action Items for next week:

1. Get a thorough understanding of the different disentangled VAE architectures and use some of them to generate the instances, and evaluate naturalness.

1. End-to-end implementation of the pipeline to see the initial results.

good results on the use of disentangled VAE. i would suggest an incremental approach. now you have some initial confidence on disentangled VAE. try to move onto conditional VAE on how to flip the sensitive attributes. try to think deep, in terms of what are the fundamental issues in this problem, what are the possible approaches to address the problem, what approaches you want to take and why, in terms of its pros/cons.

Hello Everyone,

I had a paper submission on Jan 10. I worked on the paper submission before the holidays.

For the past week, I revisited the paper for the necessary fixes mentioned by Dr. Lei. Furthermore, I reflected on this project I had undertaken since Spring 2024. I have a few slides to share regarding this project and paper submission. I want to book 30 mins of the time for the presentation.

New Action Items: I am looking forward to a new project regarding using Disentangled VAE and Conditional VAE for fairness testing.

congrats on your first submission. the two directions are indeed good ones to explore. i suggest you focus on the use of Disentangled VAE in the following week, since this is relatively clear in terms of what to do. that is, try to do some experiments to see if its use helps improve the naturalness.

Hello Everyone,

What is Completed?

Pipeline for the End-to-End Implementation.

What is pending?

Design Decisions, primarily use of the columns and how to flip the sensitive attribute.

Writeup for the Experiment, Results, and sections.

I propose two forms of processing for the dataset.

All the columns included in AI360 are used for the datasets (Adult, Compas, Credit)

1. These datasets are transformed for the t_way combination, and a VAE is trained to find the t-way combinations.

1. For training an ML model, these datasets are encoded as AI360 has transformed them ( Label Encoding: privileged group: 1, Unprivileged group: 0 ), and t_way samples are transformed accordingly to perform fairness testing.

I would like to sign up for a discussion tomorrow.

do you have a first draft of the three sections, Background, Approach, and Experiments?

Hi Group,

Milestone: Dec 20 ( Paper’s first Draft )

I have created on end-to-end implementation of the approach for one dataset ( Adult ) and obtained the results.

I will be getting results for other two datasets ( COMPAS, CREDIT ) by late afternoon to evening.

Summary of Results:

The t_way_samples are obtained from the latent space representation of the training instances.

The ATN was found to be higher when means of bin was used to generate t_way_samples. ( For t = 4, ATN: 0.775 )

Furthermore, using bin_means from discretized columns resulted in better results for the ratio of discriminatory instances compared to ratio of discriminatory instances from the “test set” of the adult income dataset.

I would like to signup for discussion for feedback on the results & further experiments.

sounds like good progress. also remember to discuss the issue of ground truth/discretization you mentioned during the meeting. please try to have an agenda and also what you expect to get out of the discussion, in terms of action items and decisions to make.

Hi All,

Milestone: Dec 24, Paper’s Internal Deadline

I have completed the following action items:

1. Discriminatory Instances are generated when a random value is selected from the bin obtained using entropy-based discretization.

Using the mean value of the bin results in a slightly better ATN.

1. Comparison with Ankita’s Approach. The current approach has better results than Ankita’s Approach.

I will present briefly the results if the time permits.

I have an implementation of Conditional VAE using Pytorch. However, I am trying to integrate the implementation with the existing library for better results.

I will complete this last action item and write the approach section as soon as possible for the design decision.

great job. now it is time to start on paper writing

Hello Everyone,

Milestone: Dec 24, Paper’s Internal Deadline

I am following up on the action items from the past week:

1. Discriminatory instances generation when a random value is selected from the bin obtained using entropy-based discretization.

1. Comparison with Ankita’s Approach.

1. Implementation of Conditional VAE.

The target date for the action items listed above was today to keep in track with the milestone; however, I was pretty busy this weekend with TA duties. I will follow up on these action items and their results via Slack, by tomorrow evening.

I would suggest you take off “possibly”. mindset matters. things happen only with commitment.

Hello Everyone,

Milestone: Nov 29: Complete Pipeline for current approach along with results

Action Items performed this week:

1. Sampling from the multi-variate normal distribution, t-way coverage selection mechanism to get the data points.

1. Compare ATN ( Ankita’s approach ) to approach (1).

Project Page: https://ajdahal.github.io/portfolio/

I will be presenting on new idea session today.

the project page does not seem to be in a good state. also please summarize the results of your action items, which would be more informative.

Hello Group,

Action Item completed this week:

I have been doing some experiments with regards to VAE for generating data points.

Fixed the issue with kl divergence for ‘hours-per-week’ observed in the last meeting.

I have obtained metrics for Reconstruction Loss vs Number of Embedding Dimension.

ATN and Kldivergence for random binning of embedding dimension.

Currently, I am carrying out t-way combination of latent space vectors. I believe I will have results by this afternoon.

Action Items for next week:

1. Complete the implementation of Conditional VAE, thus completing the pipeline.

1. Review all the Fairness definitions, and present them.

1. I have a list of papers related to my current project regarding VAE and it’s use in fairness testing. I will add state of the art papers regarding fairness testing to the list and upload them to the project page.

I would like to book 15 minutes of time for discussion on the results and the approach.

Hello Group,

I am still working on the experiments regarding fairness testing for Variational AutoEncoder.

I will share some results on slack before Nov 12 regarding the process with some metrics.

since you are targeting IWCT, whose deadline is fast approaching, it would help you can work out a schedule, with break-down tasks and target dates. the schedule will guide all your efforts toward the paper submission.

Hello Everyone,

Milestone: Baseline for Fairness Testing ( Nov 12 )

I am working on making changes on the existing library of VAE to encorporate the combinatorial sampling in it’s latent space. Furthermore, I am also looking into creating conditional VAE and recording metrics for ATN and KL divergence along the way. I am currently working on the implementation.

I will briefly discuss the findings this Friday.

Hi Group,

Milestone: Baseline for Fairness Testing

I have collected and reviewed the necessary literature for the fairness testing with variational auto-encoder and implemented the necessary code for conditional variational auto-encoder. I would like to signup for a brief discussion tomorrow regarding the review and the results.

do you want to put a target date on your milestone?

Hi Group,

Milestone: Finalize the baseline approach Oct 18

I have been working on the following action items since last week:

1. Literature review related to fairness testing using data samples that reflect realistic changes when the sensitive attribute is flipped for fairness testing.

1. Conditional VAE for data generation.

I have made some progress with the action items above, but haven’t worked till completion primarily due to travel.

I will update my progress everyday till coming Friday to ensure the baseline approach is finalized.

try your best to make the milestone

Milestone: Finalize the baseline approach for Fairness testing ( 18 / 08 )

Intermediate Goals:

A. Complete literature search for fairness testing using data samples that reflect realistic changes when sensitive attribute is flipped for fairness testing. ( 11 / 08 )

B. Generation of Data samples conditioned on specific sensitive attribute. ( 11 / 08 )

Action Items completed:

1. Compiled a list of papers that lists papers that use counterfactuals for fairness testing . Adding the description to indicate if the paper is relevant to my task.

1. I have basic code template for conditional VAE. I am debugging it for generating better samples.

I will present today in the topic of “LLM and it’s application in Tabular Data”.

sounds good. i am very interested in hearing what you found out from your literature review first, in your presentation today. also what are the specific issues with conditional VAE?

Thank You Dr. Lei for your feedback. I will be able to complete the literature review by Friday. Furthermore, I will sample few datapoints from the CTVAE and present some results this Friday.

To ensure, I make incremental progress to the task, I will update the status on channel related to my research Wednesday evening.

Hi Group,

I have been following up on action items discussed in the last meeting. I will share some findings regarding LLM producing samples of tabular data for different prompts given to it which will take around ~15 mins. I plan to continue on the action items (sampling in latent space of VAE, generating adversarial examples on top of it ) and present the results in upcoming meetings.

My next milestone would be: Sample instances in latent space of VAE, generate adversarial examples on those samples to produce discriminatory samples and analyze results in each step (Oct 4 ).

this is not a good status updates, as I cannot get useful information out of it.

Hi Group,

I have been looking into the literature regarding Fairness and Clustering since the last meeting

( shared in the channel arjun-research ). Creating Baseline approach with the findings from last few weeks is taking little more time than expected.

I intend to present the summary of findings, approach and summary of relevant literature in the upcoming meeting.

Hello All,

Milestone: Present on literature of all existing work on Fairness testing with clustering ( sep 24 )

Action items for the week: I have been following up on action items ( Divide the data into further sub-clusters to get discriminatory and non-discriminatory clusters, Inspect the datapoints that were discriminatory and non-discriminatory in labels as well as prediction ) discussed in last meeting but haven’t been able to make much progress as I amn’t very well. I will present some findings in upcoming meeting on Friday.

it is probably time to look back at what you have found out, and then start on developing your own approach. sign up a technical discussion to summarize your findings and propose some possible ideas for your own approach.

Hi All,

Milestone:

Improve upon the baseline approach

• Review all the papers regarding Fairness Testing, Active Learning

I have been investigating the clusters that have all the discriminatory samples.

I have some findings that I would like to share in tomorrow’s meeting ( ~15-20 mins ).

try to make a presentation on the major approaches to fairness testing, especially on any existing work based on clustering