Weekly Updates for Fadul Sikder

Fadul:

Hello everyone,

I’ve been a bit under the weather this week with flu-like symptoms, but I’m almost recovered. Despite this, I’ve been consolidating my research findings and plan to send them to Dr. Lei and Dr. Ji by the end of today.

Experimentation on Fine-tuning Smart Contracts

• The fine-tuning process for the smart contract model we worked on for our last paper yielded suboptimal results. To address this, I’m investigating how transformers propagate information, specifically focusing on the embeddings of the last token. Typically, the last token embedding aggregates all prior information from the sequence. A key observation is that subtle changes in smart contract—such as variations in line order ( Reentrancy)or small syntactical differences(Interger Overflow)—can introduce vulnerabilities. To assess whether the model is learning these distinctions, I plan to analyze the cosine similarity of the embeddings for different contracts by comparing the last token embeddings: Small cosinesimilarity between vectors suggests ineffective learning, indicating the model fails to distinguish between vulnerable and safe contracts and larger distances imply the model is effectively capturing discriminative patterns, critical for identifying syntactical differences that lead to vulnerabilities.

• While reviewing the literature on graph-based neural network approaches for vulnerability detection in smart contracts, I’ve noticed: no work has applied Graph Isomorphic Networks (GIN) in this domain. In theory GIN has stronger discriminative power compared to Graph Convolutional Networks (GCN) or Graph Attention Networks (GAT). One potential direction is to explore the application of GIN for vulnerability detection. Since this direction is relatively unexplored, it may present an opportunity.

Fadul:

Over the past few days, I have been reviewing literature to understand how graph-based neural networks, particularly those utilizing attention mechanisms, are being applied to smart contract vulnerability detection. Below is a summary of the methodologies I came to find:

1. Multi-Relational Nested Graph Convolutional Network (2023):

• Proposes a GNN-based method enhanced with self-attention mechanisms.

• Examines both inter-function (call graphs) and intra-function (control/data flow) relationships.

• Introduces Multi-Relational Nested Graphs to effectively locate vulnerabilities in Solidity smart contract functions.

1. VulDet: Graph Attention Networks for Vulnerability Detection (2023):

• Constructs contract graphs with node features tied to vulnerabilities.

• Utilizes Graph Attention Networks to detect vulnerabilities such as reentrancy and timestamp dependency.

1. MANDO-GURU: Heterogeneous Graph Embeddings (2022):

• Employs heterogeneous graph attention neural networks for Solidity vulnerability detection.

• Combines control-flow and call graphs to encode both structural and semantic relationships at contract and line levels.

1. Combine Sliced Joint Graph with Graph Neural Networks (2023):

• Adopts a Bidirectional Gated GNN model incorporating hybrid attention mechanisms.

• Leverages program slicing, along with AST, CFG, and PDG graphs, to improve vulnerability detection performance.

1. SCVHUNTER: Heterogeneous Graph Attention Network (2024):

• Builds semantic graphs representing smart contract features.

• Implements attention mechanisms to prioritize critical nodes, enabling detection of multiple types of vulnerabilities.

Current Focus and Next Steps:

• While I am still in the exploratory phase, I aim to narrow down the approaches and finalize key methodologies soon. The field s seems quite crowded little bit confuse which would be the best direction to take

• I am working on a proposal document detailing two potential approaches, which I plan to complete by tomorrow evening. This document will serve as a basis for a more structured discussion during our Thursday meeting.

• The ultimate goal is to identify actionable steps and align on a strategy moving forward.

Replies:

Dr. Lei:

please try to write down your findings and send to the group before our meeting on Thu

---

Fadul:

Milestone: USENIX Security ‘25

Key Dates:

• Milestone Date: January 22

• Internal Deadline: December 15-21

Weekly Updates:

1. Application of Slicing Criteria for Feature Space Reduction

I have been exploring how slicing criteria can effectively reduce the feature space. This analysis focuses on assessing its adaptability to various types of vulnerabilities.

1. Literature Review on Contrastive Learning and Graph-Based Neural Networks

• I reviewed a paper on contrastive learning (only paper that is out there for Smart Contract Vulnerability Detection with this method), which proposes a mechanism to enrich the feature space, allowing for improved learning outcomes. I am currently investigating how this approach can be adapted and integrated into my ongoing work on program slicing for vulnerability detection.

• My review of 4 papers from the past three years for GNN based Smart Contract Detection:

1. One paper highlights their GNN designs that prioritize general-purpose feature engineering, remaining largely agnostic to specific vulnerabilities.

1. The other three papers incorporate domain-specific expert knowledge into GNN architectures to enhance performance.

Replies:

Dr. Lei:

one of the two new directions we discussed is not about GNN. instead it is about GNN-Attention networks, i.e., combining GNN with Attention mechanism or pretrained LLM.

---

Fadul:

Milestone: USENIX Security ‘25

Key Dates:

• Milestone Date: January 22

• Internal Deadline: December 15-21

Tasks Completed This Week:

I have been analyzing the program slicing process with a focus on Solidity smart contracts. Building on insights from a 2024 paper that employs program slicing to detect reentrancy vulnerabilities, I aim to refine and expand its approach. Specifically, my work involves:

• Program Dependency Graph Creation:The 2024 paper begins by constructing a program dependency graph, which serves as the foundation for identifying slicing criteria.

• Defining Slicing Criteria for Reentrancy Vulnerabilities:The paper identifies specific programming statements related to external calls as the starting point. It then backtracks through the dependency graph to trace all relevant lines and their dependencies.

• Extending the Criteria to Additional Vulnerabilities:I am exploring how these slicing criteria and backtracking processes can be adapted to slice up program for a second type of vulnerability. My objective is to evaluate how much input reduction can be achieved by applying this methodology across two vulnerability definitions.

I am documenting these slicing criteria approach in details. I aim to share the document by midday. Additionally, I am implementing these criteria to assess their effectiveness in put size reduction.

Projects Page: https://fadulsikder.github.io/portfolio/

Fadul:

Milestone: USENIX Security ‘25

Key Dates:

• Milestone Date: January 22

• Internal Deadline: December 15-21

Tasks Completed This Week:

• Conducted a literature review on feature engineering research focused on Solidity smart contracts and program slicing.

• Manually created example contracts, marking lines of significance. Currently, I am exploring vulnerability-specific detection methods to do this task.

Pattern Identification in Vulnerabilities:

I can think of two ways to move forword One approach to vulnerability identification involves recognizing common patterns in abstract domains, such as Abstract Syntax Trees (AST), Control Flow Graphs (CFG), or taint analysis chains. Existing studies uses state variables or opcode-level similarities or some other form of similarities for detecting vulnerabilities. Another potential strategy is to apply a state-of-the-art approach to my data and assess its effectiveness in identifying key code segments of interest with out delving deep first.

Fadul:

Hello everyone,

Action Items Completed:

• Finished a complete draft of my fine-tuning paper on LLMs.

Next Steps: I have three main action items identified going forward:

1. Conduct experiments with a new model and configuration on a custom head.

1. Explore feature engineering approaches to refine the input space representation of smart contracts, making it more targeted.

1. Rewrite parts of my experiments with a more detailed approach to improve interpretation and debugging, especially where I encountered issues previously. I already started this rewriting phase after Friday.

Upcoming Goals: My immediate goal is to conduct a literature review and compile research on feature engineering. I plan to approach this in two phases:

• Phase 1: Meta-level paper collection and initial analysis by this Thursday.

• Phase 2: In-depth exploration, with findings ready to present by next Thursday.

While I haven’t yet decided on a specific conference date for my next Milestone, I aim to finalize this choice by my next update.

Replies:

Dr. Lei:

sounds good. one suggestion is perhaps looking beyond vulnearability detection. there is a lot of work on vulnerability detection, which is kind of crowded. are there any other problems to address with smart contracts, e.g., fault localization, contract synthesis (i.e., automatically construct a smart contract based on some kind of user description), and others?

---

Fadul:

Milestone: Submission to ISSTA on October 31st

This week, I have focused on conducting baseline experiments and setting up additional ones to observe the results. I have completed about 70% of the experiments section but am still awaiting baseline results from experiments currently running on TACC. Additionally, I have drafted the related work section, which took more time than anticipated. I haven’t updated the Overleaf document yet but plan to do so by this evening or tomorrow afternoon. Finally, I have addressed all technical comments from the last draft and adjusted the writing accordingly.

Fadul:

Next Milestone: ISSTA Paper Submission on Oct 31.

I have been working on the experiments and completing the remaining tasks. I obtained one set of results: for binary classification, an accuracy of 67%, and for multi-label classification, 48%, across a total of 17 classes with five vulnerabilities. Here is the current dataset distribution:

• safe: 3588

• ARTHM: 2487

• ARTHM+LE: 882

• LE: 580

• TimeO: 488

• ARTHM+TimeO: 346

• RENT: 313

• ARTHM+RENT: 92

• RENT+TimeO: 71

• LE+TimeO: 66

• TimeM: 62

• LE+RENT: 48

• LE+RENT+TimeO: 29

• ARTHM+RENT+TimeO: 26

• ARTHM+LE+TimeO: 26

• ARTHM+LE+RENT: 25

I believe some of the multi-label classifications are too many for the limited number of contracts available. I already reduced the original 37 labels to 17, but this still may be too many. Therefore, I aim to reduce the number of classes further and run another set of experiments with 7 labels.

The revised dataset distribution is as follows:

• safe: 3588

• ARTHM: 2487

• ARTHM+LE: 882

• LE: 580

• TimeO: 488

• ARTHM+TimeO: 346

• RENT: 313

I am currently running this new set of experiments and aim to complete the updated version by tomorrow night.

Replies:

Dr. Lei:

even though the results are not as good as expected, still try to make the submission with the results you have.

---

Fadul:

Next Major Milestone: Submit the paper to ISSTA 2025 - October 31st

Action Item Completed:

Initially, I planned to complete the environment setup for the training pipeline by Friday. However, resolving dependency conflicts proved more complex than anticipated, requiring manual inspection and management of numerous packages. I eventually completed this by Monday afternoon.

Current Experiments in Progress:

• Conducting binary classification and multi-label classification training on ScrawlID Dataset (9K Contracts) dataset

• Ablation Study: Evaluating the impact of excluding preprocessing on contracts using my approach.

• Gathering data on token lengths and GPU requirements.

• Dataset-specific Experiments: Using my composed dataset exclusively as a test set for a top-performing pre-trained model.

Dr. Lei, when you have a moment, could you please review my current draft?

Replies:

Dr. Lei:

is your latest version on overleaf? time is running out, and this is a deadline that you cannot miss. you need to have an internal deadline, say have a reasonable, complete version latest by one week before the submission.

---

Fadul:

Next Major Milestone: Submit the paper to ISSTA 2025 - October 31st

Action Items Completed:

• Reviewed the previous draft and addressed initial comments.

• Rewrote and reconnected missing scripts. This task is nearly complete, but I am still encountering some runtime errors that prevent the pipeline from running as before. Further debugging is needed.

Actions to be Completed:

• By Friday, I plan to complete the experiment setup and run the same tests on the ScrawID dataset.

• Once I obtain the results and initial analysis, I will fully define the scope of experiments to be conducted and how those can be reported into the paper.

• After receiving feedback on the current draft, I aim to complete another revision pass by next Tuesday.

Replies:

Dr. Lei:

sounds good. just want to say, time is very tight. take it as challenge to you to make the submission.

---

Fadul:

Next Milestone: October 1st – Complete the current version by October 4th.

Actions performed: Set up the experiments, but experienced a setback due to unfortunate data deletion. I am hoping to resolve this issue by today or tomorrow and get everything up and running again. Wrote the latest version addressing the comments. However, some data support and diagrams still need to be added, as suggested. I aim to finish this version by October 3rd with all the result.

Actions for the next meeting: Complete binary and multi-label classification experiments and generate other data statistics and incorporate them in paper.

Fadul:

Next Milestone: October 1st – Complete the current version.

Actions performed:

• Conducted binary classification on the old dataset.

• Written a script to scrape Solidity source code from Etherscan with SrcawlID dataset contract adresses

• With the source code, prepared a file with appropriate labels and created a CSV file to carry out experiments

Actions for the next meeting:

• Complete binary and multi-label classification experiments and present results on Friday.

• Review the draft version with necessary edits.

Replies:

Dr. Lei:

i made comments on your draft. if you want to make the milestone, you really need to work hard. there is still a lot to do for a complete version.

---

Fadul:

Milestone: October 1

Complete the current paper, including all experiments and writing.

Action Items Completed:

• Ran experiments on binary classification. Will present the results in the meeting.

• Explored potential directions for training the model with a larger dataset. These options will be discussed in the meeting.

• Prepared a presentation for a new idea.

• Completed the lab website implementation for PhD student updates.

Action Items for Next Meeting:

• Train the model with a larger dataset.

• Focus on achieving higher training accuracy while addressing the challenge of improving test accuracy. I will experiment with this.

• Write the “Related Work” section of the paper.

• Incorporate longer sequences into the fine-tuning process using distributed training.

Fadul:

added an integration to this channel: <https://softwareengin-m1a1973.slack.com/services/B07MAHUQFFD

Website Update Notification>

Fadul:

Milestone: October 1

Complete the current paper, carrying out all experiments and writing.

Action Items Before Next Meeting:

• Obtain results for binary classification.

• Train the model with a larger dataset.

Action Items Performed:

• Completed the first draft.

• Corrected data balancing implementation for training the model.

• Experimented with binary classification (results pending)

Replies:

Dr. Lei:

make a commitment to meet this milestone, whatever it takes

---