mirror of
https://github.com/karma-riuk/crab-webapp.git
synced 2025-07-05 06:08:13 +02:00
309 lines
13 KiB
HTML
309 lines
13 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="en">
|
||
|
||
<head>
|
||
<meta charset="UTF-8" />
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||
<link rel="icon" type="image/x-icon" href="/img/crab.png">
|
||
<title>Dataset Downloader & Answer Uploader</title>
|
||
<link rel="stylesheet" href="css/style.css">
|
||
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">
|
||
<script src="https://cdn.socket.io/4.5.4/socket.io.min.js"></script>
|
||
<script defer src="js/index.js"></script>
|
||
<script defer src="js/sorttable.js"></script>
|
||
<script defer src="js/modal.js"></script>
|
||
</head>
|
||
|
||
<body>
|
||
<header class="site-header">
|
||
<h3 id="page-title">
|
||
<img src="/img/crab.png" alt="Crab" class="crab-icon">
|
||
Crab Webapp
|
||
</h3>
|
||
<button id="about-button" class="info-btn">About</button>
|
||
</header>
|
||
<main>
|
||
|
||
<fieldset>
|
||
<legend>
|
||
<strong>Download a Dataset</strong>
|
||
<button id="info-download-btn" class='info-btn'><i class="fa fa-info"></i></button>
|
||
</legend>
|
||
|
||
<label for="dataset-select">Dataset:</label>
|
||
<select id="dataset-select">
|
||
<option value="comment_generation">Comment Generation</option>
|
||
<option value="code_refinement">Code Refinement</option>
|
||
</select>
|
||
<label>
|
||
<input type="checkbox" id="with-context">
|
||
Include context
|
||
</label>
|
||
<br /><br />
|
||
<button id="download-dataset">Download</button>
|
||
</fieldset>
|
||
|
||
<fieldset>
|
||
<legend>
|
||
<strong>Upload Your Answers</strong>
|
||
<button id="info-upload-btn" class='info-btn'><i class="fa fa-info"></i></button>
|
||
</legend>
|
||
<label for="answer-type">Type:</label>
|
||
<select id="answer-type">
|
||
<option value="comment">Comment Generation</option>
|
||
<option value="refinement">Code Refinement</option>
|
||
</select>
|
||
<br /><br />
|
||
<input type="file" id="file-input" accept="application/json" />
|
||
<br /><br />
|
||
<div style="display: flex; align-items: center; gap: 0.5em">
|
||
<button id="upload-btn">Upload JSON</button>
|
||
<div id="upload-status" class="hidden" style="color: green;"></div>
|
||
</div>
|
||
</fieldset>
|
||
|
||
<fieldset>
|
||
<legend>
|
||
<strong>Get status of ongoing process</strong>
|
||
<button id="info-results-btn" class='info-btn'><i class="fa fa-info"></i></button>
|
||
</legend>
|
||
<div style="display: flex; align-items: center; gap: 0.5em">
|
||
<label for="uuid">Process id:</label>
|
||
<input type="text" id="uuid" required />
|
||
</div>
|
||
<br /><br />
|
||
<button id="request-status">Request status</button>
|
||
<span id="status-status" class="hidden" style="color: green;"> </span>
|
||
</fieldset>
|
||
|
||
|
||
<div id="progress-container" class="hidden">
|
||
<h3>Processing answers...</h3>
|
||
<div>
|
||
<progress id="progress-bar" value="0" max="100"></progress>
|
||
<span id="progress-text">0%</span>
|
||
</div>
|
||
</div>
|
||
|
||
<div id="comment" class="results-container hidden">
|
||
<h3>Results Comment Generation<button class="download-results">Download</button></h3>
|
||
<table class="sortable">
|
||
<thead>
|
||
<tr>
|
||
<th>id</th>
|
||
<th>Proposed comment</th>
|
||
<th>Correct file</th>
|
||
<th>Distance</th>
|
||
<th>Max bleu score</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
|
||
<div id="refinement" class="results-container hidden">
|
||
<h3>Results Code Refinement<button class="download-results">Download</button></h3>
|
||
<table class="sortable">
|
||
<thead>
|
||
<tr>
|
||
<th>id</th>
|
||
<th>Compiled</th>
|
||
<th>Tested</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
|
||
<div id="modal-overlay" class="modal-overlay hidden" tabindex=-1>
|
||
<div class="modal-container">
|
||
<button id="modal-close" class="modal-close" aria-label="Close">×</button>
|
||
<div id="modal-content"></div>
|
||
</div>
|
||
</div>
|
||
|
||
<template id="about">
|
||
<h2>About this project</h2>
|
||
<div>
|
||
<p>
|
||
This project introduces <strong>CRAB (Code Review Automated Benchmark)</strong>, a high-quality
|
||
benchmark designed to evaluate deep learning-based code review automation tools. It focuses on two
|
||
key tasks:
|
||
</p>
|
||
<ul>
|
||
<li><strong>Comment Generation</strong>: Generating natural language review comments that identify
|
||
issues and suggest improvements for a given piece of code.</li>
|
||
<li><strong>Code Refinement</strong>: Producing revised code that correctly implements the
|
||
suggestions from a review comment.</li>
|
||
</ul>
|
||
<p>
|
||
The dataset consists of
|
||
carefully curated triplets
|
||
<code><submitted_code, reviewer_comment, revised_code></code>—ensuring each comment is
|
||
actionable and each revision implements the suggested change. This eliminates noise common in
|
||
previous datasets and supports reliable, meaningful evaluation.
|
||
</p>
|
||
<p>
|
||
To support model benchmarking, we also provide a web-based evaluation platform (the website on which
|
||
you are reading this description) that allows researchers to download the dataset, submit their
|
||
predictions, and assess model performance across both tasks.
|
||
</p>
|
||
<p>
|
||
You can explore the source code for each component here:
|
||
</p>
|
||
<ul>
|
||
<li><a href="https://github.com/karma-riuk/crab" target="_blank">Dataset Construction Repository</a>
|
||
</li>
|
||
<li><a href="https://github.com/karma-riuk/crab-webapp" target="_blank">Web App Repository</a></li>
|
||
</ul>
|
||
|
||
<p>
|
||
This website lets you evaluate code review models against the CRAB benchmark. You can download input
|
||
files for either the comment generation or code refinement task, upload your model’s predictions,
|
||
and view the results once processing is complete. Each section includes a help icon
|
||
<button class='info-btn'><i class="fa fa-info"></i></button> that provides more detailed
|
||
instructions and file format guidelines.
|
||
</p>
|
||
</div>
|
||
</template>
|
||
|
||
<template id="info-download">
|
||
<h2>Downloading the Dataset</h2>
|
||
<div>
|
||
<p>
|
||
When you download a dataset, you'll receive a ZIP archive containing a JSON file. The structure of
|
||
this file depends on the selected task.
|
||
</p>
|
||
|
||
<section class="json-schemas">
|
||
<details>
|
||
<summary><strong>Comment Generation</strong></summary>
|
||
<p>The JSON maps each ID to an object with:</p>
|
||
<ul>
|
||
<li><strong>files</strong>: a map of filenames to their content at the start of the pull
|
||
request.</li>
|
||
<li><strong>diffs</strong>: a map of filenames to the diff that was applied to each file
|
||
before
|
||
the comment was made.</li>
|
||
</ul>
|
||
<pre><code>{
|
||
"1234": {
|
||
"files": {
|
||
"src/Main.java": "public class Main { ... }"
|
||
},
|
||
"diffs": {
|
||
"src/Main.java": "@@ -1,3 +1,6 @@ ..."
|
||
}
|
||
}
|
||
}</code></pre>
|
||
</details>
|
||
|
||
<details>
|
||
<summary><strong>Code Refinement</strong></summary>
|
||
<p>The JSON structure is similar to comment generation, with one additional field:</p>
|
||
<ul>
|
||
<li><strong>files</strong>: the initial version of each file in the PR.</li>
|
||
<li><strong>diffs</strong>: the diff applied before the comment was made.</li>
|
||
<li><strong>comments</strong>: a list of comments, each with a body, the file it refers to,
|
||
and
|
||
the exact location of the comment.</li>
|
||
</ul>
|
||
<pre><code lang="json">{
|
||
"5678": {
|
||
"files": { ... },
|
||
"diffs": { ... },
|
||
"comments": [
|
||
{
|
||
"body": "Consider simplifying this logic.",
|
||
"file": "src/Util.java",
|
||
"location": {
|
||
"start_line": 42,
|
||
"end_line": 45
|
||
}
|
||
}
|
||
]
|
||
}
|
||
}</code></pre>
|
||
</details>
|
||
</section>
|
||
|
||
<h3>With Context (Optional)</h3>
|
||
<p>
|
||
You can choose to download the dataset with full repository context — the state of the entire
|
||
codebase at the time the PR was created. This may help your model better understand the broader
|
||
project structure and dependencies outside of the changed files.
|
||
</p>
|
||
</div>
|
||
</template>
|
||
<template id="info-upload">
|
||
<h2>Uploading Results</h2>
|
||
<section>
|
||
<p>
|
||
After downloading a dataset and generating your predictions for either task, you can upload your
|
||
results here to start the evaluation process.
|
||
</p>
|
||
<p>
|
||
Your uploaded JSON file must follow one of the schemas described below, depending on the selected
|
||
task. Once uploaded, the system will begin evaluating your submission. A progress bar will appear to
|
||
show how far along the evaluation is.
|
||
</p>
|
||
<p>
|
||
For <strong>Code Refinement</strong>, an id will also be displayed. This id allows you to safely
|
||
close the browser tab and later check the evaluation progress by pasting it into the <em>"Get
|
||
status of ongoing process"</em> section. More information is available in that section.
|
||
</p>
|
||
<section class="json-schemas">
|
||
<details>
|
||
<summary>
|
||
<strong>Comment Generation</strong>
|
||
</summary>
|
||
<p>
|
||
Submit a JSON object where each key is a string ID and each value is the generated comment.
|
||
</p>
|
||
<pre><code>
|
||
{
|
||
"1234": "This method lacks null checks.",
|
||
"5678": "Consider renaming this variable for clarity."
|
||
}
|
||
</code></pre>
|
||
|
||
</details>
|
||
<details>
|
||
<summary><strong>Code Refinement</strong></summary>
|
||
<p>
|
||
Submit a JSON object where each key is a string ID, and the value is another object that
|
||
maps
|
||
file
|
||
paths (relative to the top-level directory) to the new file content.
|
||
</p>
|
||
<pre><code>
|
||
{
|
||
"1234": {
|
||
"src/Main.java": "public class Main { /* updated code */ }"
|
||
},
|
||
"5678": {
|
||
"utils/Helper.java": "public class Helper { /* improved logic */ }"
|
||
}
|
||
}
|
||
</code></pre>
|
||
|
||
<p>
|
||
Make sure your file strictly follows the expected format to avoid upload errors.
|
||
</p>
|
||
</details>
|
||
</section>
|
||
</section>
|
||
</template>
|
||
<template id="info-results">
|
||
<h2>Getting information of ongoing process</h2>
|
||
<div>
|
||
yes
|
||
</div>
|
||
</template>
|
||
</main>
|
||
</body>
|
||
|
||
</html>
|