Fadul:
This week, my main focus has been on training a graph neural network (GNN) for vulnerability classification. I have been reviewing relevant literature and exploring existing repositories that implement GNN-based models. From my investigation, it appears that reproducing full research pipelines from these repositories is more complex than building a GNN from scratch, so I am currently working on developing a baseline implementation.
In parallel, I have been studying Professor Ji’s Illuminati paper and analyzing its associated codebase to understand how its methods could be adapted into my own work.
I have defined the following three short-term goals:
My aim is to make progress on all three items before Thursday meeting.
Fadul:
Weekly Research Update
This week, I have been focusing on implementing graph neural networks (GNNs) and exploring different methods of transforming a program into a graph representation. One approach I identified involves first converting the program into a relational database and then constructing the graph from that data. I am currently evaluating the validity and effectiveness of this approach.
My main objective is to build a GNN model capable of making line-level vulnerability predictions and train it to improve performance. Additionally, I have been reviewing repositories that utilize GNNExplainer to better understand their methodologies and applications.
I have been defining specific action items for the next steps of my project. My plan is to:
Finalize and refine these action items.
Share them before Thursday’s meeting for discussion and feedback.
I have also prepared for my new idea presentation this week, focusing on the DeepSeekMath paper, which introduced Group Relative Policy Optimization (GRPO). GRPO is a reinforcement learning-based post-training method designed to improve reasoning capabilities in large language models (LLMs).
My preliminary idea explores whether GRPO can be applied to symbolic execution to enhance program analysis. While I haven’t found prior work explicitly applying GRPO in this domain, I am conducting a deeper analysis to assess its feasibility. This will be the key topic of discussion in my new idea session next week.
Continue refining my GNN implementation and validate the relational database approach.
Post the finalized action items before Thursday’s meeting.
Replies:
Dr. Lei:
before thursday’s meeting, try to think more about the topic we discussed earlier, i.e., how to take explanations produced by GNNExplainer or other types of explanations to produce actual test cases to verify if the explanations are valid.
in the future, please prepare slides for technical discussions.
Fadul:
This week, I submitted a paper for PROMISE 2025 on Saturday morning, which kept me occupied with the submission process. After that, I began exploring research papers related to a new research idea I am considering. My focus is on developing a project that post-trains LLMs with reinforcement learning for security analysis using symbolic execution. I plan to present this idea and discuss relevant papers in my new idea session next week.
Research Directions
Next Steps
For both of these research directions, my immediate task is to outline concrete action items. I aim to have a draft and post them by tonight. Once posted, I will review and discuss them further to determine which direction to prioritize as my primary focus.
Replies:
Dr. Lei:
first, congrats on your first submission! the new research ideas look good. try to do some reflection on the past project and share with the group.
Fadul:
Hello everyone,
This week, I have been focused on finalizing my research paper. Additionally, I trained more models, as we discussed in our meetings, to provide a more comprehensive analysis of how freezing different layers affects LLM performance.
Initially, I estimated that I would complete the paper by Friday night. However, as I have been piecing together the writing, I realized that I had underestimated the timeline. Specifically, I spent a significant amount of time on the Introduction and Related Work sections and mappings of their citations—for better or worse. I am now aiming to complete a version by tonight.
—Fadul
Replies:
Dr. Lei:
it is not uncommon to underestimate the timeline. this is why it is important to budget some buffer time; otherwise, you can easily miss some deadlines. also, it is important for you to respond to emails timely. i think dr. Ji is waiting for your response as of now.
Fadul:
This week, I primarily focused on running experiments. I have obtained some partial results, including training the LLaMA 3.2 model with specific layer-freezing configurations that we have been discussing over the past few days. Additionally, I am running experiments on other models and expect to receive their results later this week.
Aside from experiments, I have also been working on submissions. The abstract submission is due tonight, and I have prepared an initial draft, which I will be sending to Dr. Lei and Dr. Ji within the next few hours. In today’s meeting, I plan to discuss these results in details.
Paper Submission Plan
With the paper submission deadline set for next Tuesday, my plan is as follows:
Replies:
Dr. Lei:
sounds good. try your best to make your first submission
Fadul:
This week, I focused on training and fine-tuning models by selectively unfreezing different layers. Some initial results were presented on Thursday. Following further discussions on Thursday and Friday, I reevaluated how to approach the research questions.
As part of this process, I conducted a literature review to better understand existing methods for layer-specific fine-tuning. I compiled my findings into a PDF, which I will share with Professor Ji and Professor Lei via email.
One key takeaway from my research is that layer-specific fine-tuning is already well-studied in existing work. As a result, I have to reconsider how to position my research contribution. Rather than framing it as a new research question, I need to refine the paper’s scope to present it in a more general yet impactful way.
My next steps involve completing the experiments and go over the paper the previous draft of the paper by Wednesday night.
Replies:
Dr. Lei:
challenge yourself to make your first submission on time. if possible try to include more LLMs in your experiments. but focus on one LLM at a time
Fadul:
Hello Everyone,
This week, I have been running several experiments that were planned based on our discussions in Thursday’s meeting. Additionally, I am working on the experimental design and formulating the rationale behind key decisions.
Additionally, I am conducting small-scale experiments on fine-tuning different layers of the LLaMA 3.2 model. Currently, I am in the process of gathering and analyzing the results.
My primary goal is to complete the experiments related to the first two research questions and report the results. In parallel, I aim to finalize the experimental design and decision-making section by Wednesday night.
Replies:
Dr. Lei:
in your writeup, try to clearly formulate the research question(s), a high-level plan to investigate the research questions, and then specific design decisions you make with justifications (w.r.t. the research questions/high-level plan).
also, it would be best to do the experiments for more than one open-source LLM if possible.
Fadul:
Milestone: Promise for February 24
Action Items and Updates:
Implementation Review and Debugging:
This week, I re-ran the last set of experiments from the previous framework using a new implementation. However, the results obtained so far are inconsistent and do not make sense. This indicates that further investigation is required to ensure the implementation is correct. My immediate focus is to debug and refine the implementation to achieve meaningful results.
Exploration of Research Questions:
Based on Professor Lei’s suggestions, I have identified and categorized three key research questions for further exploration:
Next Steps:
My goal is to complete as much of the experiments as possible and present preliminary results by Wednesday night.
Replies:
Dr. Lei:
considering the time constraint, it is important to prioritize, in terms of what would be most important for you to make the submission. for example experiment 3 is probably not important for this paper, unless you can do it very quickly.
Fadul:
Action Items, Updates, and Paper Submission Plan:
Exploring the Use of GNNExplainer for GNN-Based Vulnerability Detection
Contrastive Learning Paper Insights
PROMISE 2025 Submission Plan for Fine-Tuning LLM Paper
Current Context:
Paper Submission Plan:
Replies:
Dr. Lei:
extremely well-written report. this sets an example for how a weekly report to be written. great job!
one suggestion: between now and the submission deadline, your top priority should be on the PROMISE submission. make a commitment: do whatever it takes to make it. this will be our last effort on this project. so don’t leave any regret, i.e., try to do whatever you can reasonably think of to save this project.
Fadul:
Milestone: USENIX Security ‘25
Key Dates:
Milestone Date: January 22
Completed Tasks:
Tasks to Do:
Replies:
Dr. Lei:
very good. i think the writeups you are providing are really good and help make the discussions more productive. keep it up.
Fadul:
Hello everyone,
I’ve been a bit under the weather this week with flu-like symptoms, but I’m almost recovered. Despite this, I’ve been consolidating my research findings and plan to send them to Dr. Lei and Dr. Ji by the end of today.
Experimentation on Fine-tuning Smart Contracts
Fadul:
Over the past few days, I have been reviewing literature to understand how graph-based neural networks, particularly those utilizing attention mechanisms, are being applied to smart contract vulnerability detection. Below is a summary of the methodologies I came to find:
Current Focus and Next Steps:
Replies:
Dr. Lei:
please try to write down your findings and send to the group before our meeting on Thu
Fadul:
Milestone: USENIX Security ‘25
Key Dates:
Weekly Updates:
I have been exploring how slicing criteria can effectively reduce the feature space. This analysis focuses on assessing its adaptability to various types of vulnerabilities.
Replies:
Dr. Lei:
one of the two new directions we discussed is not about GNN. instead it is about GNN-Attention networks, i.e., combining GNN with Attention mechanism or pretrained LLM.
Fadul:
Milestone: USENIX Security ‘25
Key Dates:
Tasks Completed This Week:
I have been analyzing the program slicing process with a focus on Solidity smart contracts. Building on insights from a 2024 paper that employs program slicing to detect reentrancy vulnerabilities, I aim to refine and expand its approach. Specifically, my work involves:
I am documenting these slicing criteria approach in details. I aim to share the document by midday. Additionally, I am implementing these criteria to assess their effectiveness in put size reduction.
Projects Page: https://fadulsikder.github.io/portfolio/
Fadul:
Milestone: USENIX Security ‘25
Key Dates:
Tasks Completed This Week:
Pattern Identification in Vulnerabilities:
I can think of two ways to move forword One approach to vulnerability identification involves recognizing common patterns in abstract domains, such as Abstract Syntax Trees (AST), Control Flow Graphs (CFG), or taint analysis chains. Existing studies uses state variables or opcode-level similarities or some other form of similarities for detecting vulnerabilities. Another potential strategy is to apply a state-of-the-art approach to my data and assess its effectiveness in identifying key code segments of interest with out delving deep first.
Fadul:
Hello everyone,
Action Items Completed:
Next Steps: I have three main action items identified going forward:
Upcoming Goals: My immediate goal is to conduct a literature review and compile research on feature engineering. I plan to approach this in two phases:
While I haven’t yet decided on a specific conference date for my next Milestone, I aim to finalize this choice by my next update.
Replies:
Dr. Lei:
sounds good. one suggestion is perhaps looking beyond vulnearability detection. there is a lot of work on vulnerability detection, which is kind of crowded. are there any other problems to address with smart contracts, e.g., fault localization, contract synthesis (i.e., automatically construct a smart contract based on some kind of user description), and others?
Fadul:
Milestone: Submission to ISSTA on October 31st
This week, I have focused on conducting baseline experiments and setting up additional ones to observe the results. I have completed about 70% of the experiments section but am still awaiting baseline results from experiments currently running on TACC. Additionally, I have drafted the related work section, which took more time than anticipated. I haven’t updated the Overleaf document yet but plan to do so by this evening or tomorrow afternoon. Finally, I have addressed all technical comments from the last draft and adjusted the writing accordingly.
Fadul:
Next Milestone: ISSTA Paper Submission on Oct 31.
I have been working on the experiments and completing the remaining tasks. I obtained one set of results: for binary classification, an accuracy of 67%, and for multi-label classification, 48%, across a total of 17 classes with five vulnerabilities. Here is the current dataset distribution:
I believe some of the multi-label classifications are too many for the limited number of contracts available. I already reduced the original 37 labels to 17, but this still may be too many. Therefore, I aim to reduce the number of classes further and run another set of experiments with 7 labels.
The revised dataset distribution is as follows:
I am currently running this new set of experiments and aim to complete the updated version by tomorrow night.
Replies:
Dr. Lei:
even though the results are not as good as expected, still try to make the submission with the results you have.
Fadul:
Next Major Milestone: Submit the paper to ISSTA 2025 - October 31st
Action Item Completed:
Initially, I planned to complete the environment setup for the training pipeline by Friday. However, resolving dependency conflicts proved more complex than anticipated, requiring manual inspection and management of numerous packages. I eventually completed this by Monday afternoon.
Current Experiments in Progress:
Dr. Lei, when you have a moment, could you please review my current draft?
Replies:
Dr. Lei:
is your latest version on overleaf? time is running out, and this is a deadline that you cannot miss. you need to have an internal deadline, say have a reasonable, complete version latest by one week before the submission.
Fadul:
Next Major Milestone: Submit the paper to ISSTA 2025 - October 31st
Action Items Completed:
Actions to be Completed:
Replies:
Dr. Lei:
sounds good. just want to say, time is very tight. take it as challenge to you to make the submission.
Fadul:
Next Milestone: October 1st – Complete the current version by October 4th.
Actions performed: Set up the experiments, but experienced a setback due to unfortunate data deletion. I am hoping to resolve this issue by today or tomorrow and get everything up and running again. Wrote the latest version addressing the comments. However, some data support and diagrams still need to be added, as suggested. I aim to finish this version by October 3rd with all the result.
Actions for the next meeting: Complete binary and multi-label classification experiments and generate other data statistics and incorporate them in paper.
Fadul:
Next Milestone: October 1st – Complete the current version.
Actions performed:
Actions for the next meeting:
Replies:
Dr. Lei:
i made comments on your draft. if you want to make the milestone, you really need to work hard. there is still a lot to do for a complete version.
Fadul:
Milestone: October 1
Complete the current paper, including all experiments and writing.
Action Items Completed:
Action Items for Next Meeting:
Fadul:
added an integration to this channel: <https://softwareengin-m1a1973.slack.com/services/B07MAHUQFFD | Website Update Notification> |
Fadul:
Milestone: October 1
Complete the current paper, carrying out all experiments and writing.
Action Items Before Next Meeting:
Action Items Performed:
Replies:
Dr. Lei:
make a commitment to meet this milestone, whatever it takes
© 2023 Jeff Lei's Lab.
Site made with Jekyll. Template from the Allan Lab
We are part of the CSE Department at University of Texas at Arlington.