authorgravatar for mlugg@mlugg.co.ukMatthew Lugg <mlugg@mlugg.co.uk> 2025-07-10 09:18:10+01:00
committergravatar for mlugg@mlugg.co.ukMatthew Lugg <mlugg@mlugg.co.uk> 2025-08-01 23:48:21+01:00
logdcc3e6e1dd224f1719b0ad9ef6d8d9dc0ed497ec
tree470bd1fc20bdbeb6e4212bd4cbbd694401d757e7
parenta00edbd52d03645366c165e860d9e0ab89caa2fc

build system: replace fuzzing UI with build UI, add time report

This commit replaces the "fuzzer" UI, previously accessed with the `--fuzz` and `--port` flags, with a more interesting web UI which allows more interactions with the Zig build system. Most notably, it allows accessing the data emitted by a new "time report" system, which allows users to see which parts of Zig programs take the longest to compile. The option to expose the web UI is `--webui`. By default, it will listen on `[::1]` on a random port, but any IPv6 or IPv4 address can be specified with e.g. `--webui=[::1]:8000` or `--webui=127.0.0.1:8000`. The options `--fuzz` and `--time-report` both imply `--webui` if not given. Currently, `--webui` is incompatible with `--watch`; specifying both will cause `zig build` to exit with a fatal error. When the web UI is enabled, the build runner spawns the web server as soon as the configure phase completes. The frontend code consists of one HTML file, one JavaScript file, two CSS files, and a few Zig source files which are built into a WASM blob on-demand -- this is all very similar to the old fuzzer UI. Also inherited from the fuzzer UI is that the build system communicates with web clients over a WebSocket connection. When the build finishes, if `--webui` was passed (i.e. if the web server is running), the build runner does not terminate; it continues running to serve web requests, allowing interactive control of the build system. In the web interface is an overall "status" indicating whether a build is currently running, and also a list of all steps in this build. There are visual indicators (colors and spinners) for in-progress, succeeded, and failed steps. There is a "Rebuild" button which will cause the build system to reset the state of every step (note that this does not affect caching) and evaluate the step graph again. If `--time-report` is passed to `zig build`, a new section of the interface becomes visible, which associates every build step with a "time report". For most steps, this is just a simple "time taken" value. However, for `Compile` steps, the compiler communicates with the build system to provide it with much more interesting information: time taken for various pipeline phases, with a per-declaration and per-file breakdown, sorted by slowest declarations/files first. This feature is still in its early stages: the data can be a little tricky to understand, and there is no way to, for instance, sort by different properties, or filter to certain files. However, it has already given us some interesting statistics, and can be useful for spotting, for instance, particularly complex and slow compile-time logic. Additionally, if a compilation uses LLVM, its time report includes the "LLVM pass timing" information, which was previously accessible with the (now removed) `-ftime-report` compiler flag. To make time reports more useful, ZIR and compilation caches are ignored by the Zig compiler when they are enabled -- in other words, `Compile` steps *always* run, even if their result should be cached. This means that the flag can be used to analyze a project's compile time without having to repeatedly clear cache directory, for instance. However, when using `-fincremental`, updates other than the first will only show you the statistics for what changed on that particular update. Notably, this gives us a fairly nice way to see exactly which declarations were re-analyzed by an incremental update. If `--fuzz` is passed to `zig build`, another section of the web interface becomes visible, this time exposing the fuzzer. This is quite similar to the fuzzer UI this commit replaces, with only a few cosmetic tweaks. The interface is closer than before to supporting multiple fuzz steps at a time (in line with the overall strategy for this build UI, the goal will be for all of the fuzz steps to be accessible in the same interface), but still doesn't actually support it. The fuzzer UI looks quite different under the hood: as a result, various bugs are fixed, although other bugs remain. For instance, viewing the source code of any file other than the root of the main module is completely broken (as on master) due to some bogus file-to-module assignment logic in the fuzzer UI. Implementation notes: * The `lib/build-web/` directory holds the client side of the web UI. * The general server logic is in `std.Build.WebServer`. * Fuzzing-specific logic is in `std.Build.Fuzz`. * `std.Build.abi` is the new home of `std.Build.Fuzz.abi`, since it now relates to the build system web UI in general. * The build runner now has an **actual** general-purpose allocator, because thanks to `--watch` and `--webui`, the process can be arbitrarily long-lived. The gpa is `std.heap.DebugAllocator`, but the arena remains backed by `std.heap.page_allocator` for efficiency. I fixed several crashes caused by conflation of `gpa` and `arena` in the build runner and `std.Build`, but there may still be some I have missed. * The I/O logic in `std.Build.WebServer` is pretty gnarly; there are a *lot* of threads involved. I anticipate this situation improving significantly once the `std.Io` interface (with concurrency support) is introduced.

39 files changed, 3914 insertions(+), 1950 deletions(-)

lib/build-web/fuzz.zig created+377
......@@ -0,0 +1,377 @@
1// Server timestamp.
2var start_fuzzing_timestamp: i64 = undefined;
3
4const js = struct {
5 extern "fuzz" fn requestSources() void;
6 extern "fuzz" fn ready() void;
7
8 extern "fuzz" fn updateStats(html_ptr: [*]const u8, html_len: usize) void;
9 extern "fuzz" fn updateEntryPoints(html_ptr: [*]const u8, html_len: usize) void;
10 extern "fuzz" fn updateSource(html_ptr: [*]const u8, html_len: usize) void;
11 extern "fuzz" fn updateCoverage(covered_ptr: [*]const SourceLocationIndex, covered_len: u32) void;
12};
13
14pub fn sourceIndexMessage(msg_bytes: []u8) error{OutOfMemory}!void {
15 Walk.files.clearRetainingCapacity();
16 Walk.decls.clearRetainingCapacity();
17 Walk.modules.clearRetainingCapacity();
18 recent_coverage_update.clearRetainingCapacity();
19 selected_source_location = null;
20
21 js.requestSources();
22
23 const Header = abi.fuzz.SourceIndexHeader;
24 const header: Header = @bitCast(msg_bytes[0..@sizeOf(Header)].*);
25
26 const directories_start = @sizeOf(Header);
27 const directories_end = directories_start + header.directories_len * @sizeOf(Coverage.String);
28 const files_start = directories_end;
29 const files_end = files_start + header.files_len * @sizeOf(Coverage.File);
30 const source_locations_start = files_end;
31 const source_locations_end = source_locations_start + header.source_locations_len * @sizeOf(Coverage.SourceLocation);
32 const string_bytes = msg_bytes[source_locations_end..][0..header.string_bytes_len];
33
34 const directories: []const Coverage.String = @alignCast(std.mem.bytesAsSlice(Coverage.String, msg_bytes[directories_start..directories_end]));
35 const files: []const Coverage.File = @alignCast(std.mem.bytesAsSlice(Coverage.File, msg_bytes[files_start..files_end]));
36 const source_locations: []const Coverage.SourceLocation = @alignCast(std.mem.bytesAsSlice(Coverage.SourceLocation, msg_bytes[source_locations_start..source_locations_end]));
37
38 start_fuzzing_timestamp = header.start_timestamp;
39 try updateCoverageSources(directories, files, source_locations, string_bytes);
40 js.ready();
41}
42
43var coverage = Coverage.init;
44/// Index of type `SourceLocationIndex`.
45var coverage_source_locations: std.ArrayListUnmanaged(Coverage.SourceLocation) = .empty;
46/// Contains the most recent coverage update message, unmodified.
47var recent_coverage_update: std.ArrayListAlignedUnmanaged(u8, .of(u64)) = .empty;
48
49fn updateCoverageSources(
50 directories: []const Coverage.String,
51 files: []const Coverage.File,
52 source_locations: []const Coverage.SourceLocation,
53 string_bytes: []const u8,
54) !void {
55 coverage.directories.clearRetainingCapacity();
56 coverage.files.clearRetainingCapacity();
57 coverage.string_bytes.clearRetainingCapacity();
58 coverage_source_locations.clearRetainingCapacity();
59
60 try coverage_source_locations.appendSlice(gpa, source_locations);
61 try coverage.string_bytes.appendSlice(gpa, string_bytes);
62
63 try coverage.files.entries.resize(gpa, files.len);
64 @memcpy(coverage.files.entries.items(.key), files);
65 try coverage.files.reIndexContext(gpa, .{ .string_bytes = coverage.string_bytes.items });
66
67 try coverage.directories.entries.resize(gpa, directories.len);
68 @memcpy(coverage.directories.entries.items(.key), directories);
69 try coverage.directories.reIndexContext(gpa, .{ .string_bytes = coverage.string_bytes.items });
70}
71
72pub fn coverageUpdateMessage(msg_bytes: []u8) error{OutOfMemory}!void {
73 recent_coverage_update.clearRetainingCapacity();
74 recent_coverage_update.appendSlice(gpa, msg_bytes) catch @panic("OOM");
75 try updateStats();
76 try updateCoverage();
77}
78
79var entry_points: std.ArrayListUnmanaged(SourceLocationIndex) = .empty;
80
81pub fn entryPointsMessage(msg_bytes: []u8) error{OutOfMemory}!void {
82 const header: abi.fuzz.EntryPointHeader = @bitCast(msg_bytes[0..@sizeOf(abi.fuzz.EntryPointHeader)].*);
83 const slis: []align(1) const SourceLocationIndex = @ptrCast(msg_bytes[@sizeOf(abi.fuzz.EntryPointHeader)..]);
84 assert(slis.len == header.locsLen());
85 try entry_points.resize(gpa, slis.len);
86 @memcpy(entry_points.items, slis);
87 try updateEntryPoints();
88}
89
90/// Index into `coverage_source_locations`.
91const SourceLocationIndex = enum(u32) {
92 _,
93
94 fn haveCoverage(sli: SourceLocationIndex) bool {
95 return @intFromEnum(sli) < coverage_source_locations.items.len;
96 }
97
98 fn ptr(sli: SourceLocationIndex) *Coverage.SourceLocation {
99 return &coverage_source_locations.items[@intFromEnum(sli)];
100 }
101
102 fn sourceLocationLinkHtml(
103 sli: SourceLocationIndex,
104 out: *std.ArrayListUnmanaged(u8),
105 focused: bool,
106 ) Allocator.Error!void {
107 const sl = sli.ptr();
108 try out.writer(gpa).print("<code{s}>", .{
109 @as([]const u8, if (focused) " class=\"status-running\"" else ""),
110 });
111 try sli.appendPath(out);
112 try out.writer(gpa).print(":{d}:{d} </code><button class=\"linkish\" onclick=\"wasm_exports.fuzzSelectSli({d});\">View</button>", .{
113 sl.line,
114 sl.column,
115 @intFromEnum(sli),
116 });
117 }
118
119 fn appendPath(sli: SourceLocationIndex, out: *std.ArrayListUnmanaged(u8)) Allocator.Error!void {
120 const sl = sli.ptr();
121 const file = coverage.fileAt(sl.file);
122 const file_name = coverage.stringAt(file.basename);
123 const dir_name = coverage.stringAt(coverage.directories.keys()[file.directory_index]);
124 try html_render.appendEscaped(out, dir_name);
125 try out.appendSlice(gpa, "/");
126 try html_render.appendEscaped(out, file_name);
127 }
128
129 fn toWalkFile(sli: SourceLocationIndex) ?Walk.File.Index {
130 var buf: std.ArrayListUnmanaged(u8) = .empty;
131 defer buf.deinit(gpa);
132 sli.appendPath(&buf) catch @panic("OOM");
133 return @enumFromInt(Walk.files.getIndex(buf.items) orelse return null);
134 }
135
136 fn fileHtml(
137 sli: SourceLocationIndex,
138 out: *std.ArrayListUnmanaged(u8),
139 ) error{ OutOfMemory, SourceUnavailable }!void {
140 const walk_file_index = sli.toWalkFile() orelse return error.SourceUnavailable;
141 const root_node = walk_file_index.findRootDecl().get().ast_node;
142 var annotations: std.ArrayListUnmanaged(html_render.Annotation) = .empty;
143 defer annotations.deinit(gpa);
144 try computeSourceAnnotations(sli.ptr().file, walk_file_index, &annotations, coverage_source_locations.items);
145 html_render.fileSourceHtml(walk_file_index, out, root_node, .{
146 .source_location_annotations = annotations.items,
147 }) catch |err| {
148 fatal("unable to render source: {s}", .{@errorName(err)});
149 };
150 }
151};
152
153fn computeSourceAnnotations(
154 cov_file_index: Coverage.File.Index,
155 walk_file_index: Walk.File.Index,
156 annotations: *std.ArrayListUnmanaged(html_render.Annotation),
157 source_locations: []const Coverage.SourceLocation,
158) !void {
159 // Collect all the source locations from only this file into this array
160 // first, then sort by line, col, so that we can collect annotations with
161 // O(N) time complexity.
162 var locs: std.ArrayListUnmanaged(SourceLocationIndex) = .empty;
163 defer locs.deinit(gpa);
164
165 for (source_locations, 0..) |sl, sli_usize| {
166 if (sl.file != cov_file_index) continue;
167 const sli: SourceLocationIndex = @enumFromInt(sli_usize);
168 try locs.append(gpa, sli);
169 }
170
171 std.mem.sortUnstable(SourceLocationIndex, locs.items, {}, struct {
172 pub fn lessThan(context: void, lhs: SourceLocationIndex, rhs: SourceLocationIndex) bool {
173 _ = context;
174 const lhs_ptr = lhs.ptr();
175 const rhs_ptr = rhs.ptr();
176 if (lhs_ptr.line < rhs_ptr.line) return true;
177 if (lhs_ptr.line > rhs_ptr.line) return false;
178 return lhs_ptr.column < rhs_ptr.column;
179 }
180 }.lessThan);
181
182 const source = walk_file_index.get_ast().source;
183 var line: usize = 1;
184 var column: usize = 1;
185 var next_loc_index: usize = 0;
186 for (source, 0..) |byte, offset| {
187 if (byte == '\n') {
188 line += 1;
189 column = 1;
190 } else {
191 column += 1;
192 }
193 while (true) {
194 if (next_loc_index >= locs.items.len) return;
195 const next_sli = locs.items[next_loc_index];
196 const next_sl = next_sli.ptr();
197 if (next_sl.line > line or (next_sl.line == line and next_sl.column >= column)) break;
198 try annotations.append(gpa, .{
199 .file_byte_offset = offset,
200 .dom_id = @intFromEnum(next_sli),
201 });
202 next_loc_index += 1;
203 }
204 }
205}
206
207export fn fuzzUnpackSources(tar_ptr: [*]u8, tar_len: usize) void {
208 const tar_bytes = tar_ptr[0..tar_len];
209 log.debug("received {d} bytes of sources.tar", .{tar_bytes.len});
210
211 unpackSourcesInner(tar_bytes) catch |err| {
212 fatal("unable to unpack sources.tar: {s}", .{@errorName(err)});
213 };
214}
215
216fn unpackSourcesInner(tar_bytes: []u8) !void {
217 var tar_reader: std.Io.Reader = .fixed(tar_bytes);
218 var file_name_buffer: [1024]u8 = undefined;
219 var link_name_buffer: [1024]u8 = undefined;
220 var it: std.tar.Iterator = .init(&tar_reader, .{
221 .file_name_buffer = &file_name_buffer,
222 .link_name_buffer = &link_name_buffer,
223 });
224 while (try it.next()) |tar_file| {
225 switch (tar_file.kind) {
226 .file => {
227 if (tar_file.size == 0 and tar_file.name.len == 0) break;
228 if (std.mem.endsWith(u8, tar_file.name, ".zig")) {
229 log.debug("found file: '{s}'", .{tar_file.name});
230 const file_name = try gpa.dupe(u8, tar_file.name);
231 if (std.mem.indexOfScalar(u8, file_name, '/')) |pkg_name_end| {
232 const pkg_name = file_name[0..pkg_name_end];
233 const gop = try Walk.modules.getOrPut(gpa, pkg_name);
234 const file: Walk.File.Index = @enumFromInt(Walk.files.entries.len);
235 if (!gop.found_existing or
236 std.mem.eql(u8, file_name[pkg_name_end..], "/root.zig") or
237 std.mem.eql(u8, file_name[pkg_name_end + 1 .. file_name.len - ".zig".len], pkg_name))
238 {
239 gop.value_ptr.* = file;
240 }
241 const file_bytes = tar_reader.take(@intCast(tar_file.size)) catch unreachable;
242 it.unread_file_bytes = 0; // we have read the whole thing
243 assert(file == try Walk.add_file(file_name, file_bytes));
244 }
245 } else {
246 log.warn("skipping: '{s}' - the tar creation should have done that", .{tar_file.name});
247 }
248 },
249 else => continue,
250 }
251 }
252}
253
254fn updateStats() error{OutOfMemory}!void {
255 @setFloatMode(.optimized);
256
257 if (recent_coverage_update.items.len == 0) return;
258
259 const hdr: *abi.fuzz.CoverageUpdateHeader = @alignCast(@ptrCast(
260 recent_coverage_update.items[0..@sizeOf(abi.fuzz.CoverageUpdateHeader)],
261 ));
262
263 const covered_src_locs: usize = n: {
264 var n: usize = 0;
265 const covered_bits = recent_coverage_update.items[@sizeOf(abi.fuzz.CoverageUpdateHeader)..];
266 for (covered_bits) |byte| n += @popCount(byte);
267 break :n n;
268 };
269 const total_src_locs = coverage_source_locations.items.len;
270
271 const avg_speed: f64 = speed: {
272 const ns_elapsed: f64 = @floatFromInt(nsSince(start_fuzzing_timestamp));
273 const n_runs: f64 = @floatFromInt(hdr.n_runs);
274 break :speed n_runs / (ns_elapsed / std.time.ns_per_s);
275 };
276
277 const html = try std.fmt.allocPrint(gpa,
278 \\<span slot="stat-total-runs">{d}</span>
279 \\<span slot="stat-unique-runs">{d} ({d:.1}%)</span>
280 \\<span slot="stat-coverage">{d} / {d} ({d:.1}%)</span>
281 \\<span slot="stat-speed">{d:.0}</span>
282 , .{
283 hdr.n_runs,
284 hdr.unique_runs,
285 @as(f64, @floatFromInt(hdr.unique_runs)) / @as(f64, @floatFromInt(hdr.n_runs)),
286 covered_src_locs,
287 total_src_locs,
288 @as(f64, @floatFromInt(covered_src_locs)) / @as(f64, @floatFromInt(total_src_locs)),
289 avg_speed,
290 });
291 defer gpa.free(html);
292
293 js.updateStats(html.ptr, html.len);
294}
295
296fn updateEntryPoints() error{OutOfMemory}!void {
297 var html: std.ArrayListUnmanaged(u8) = .empty;
298 defer html.deinit(gpa);
299 for (entry_points.items) |sli| {
300 try html.appendSlice(gpa, "<li>");
301 try sli.sourceLocationLinkHtml(&html, selected_source_location == sli);
302 try html.appendSlice(gpa, "</li>\n");
303 }
304 js.updateEntryPoints(html.items.ptr, html.items.len);
305}
306
307fn updateCoverage() error{OutOfMemory}!void {
308 if (recent_coverage_update.items.len == 0) return;
309 const want_file = (selected_source_location orelse return).ptr().file;
310
311 var covered: std.ArrayListUnmanaged(SourceLocationIndex) = .empty;
312 defer covered.deinit(gpa);
313
314 // This code assumes 64-bit elements, which is incorrect if the executable
315 // being fuzzed is not a 64-bit CPU. It also assumes little-endian which
316 // can also be incorrect.
317 comptime assert(abi.fuzz.CoverageUpdateHeader.trailing[0] == .pc_bits_usize);
318 const n_bitset_elems = (coverage_source_locations.items.len + @bitSizeOf(u64) - 1) / @bitSizeOf(u64);
319 const covered_bits = std.mem.bytesAsSlice(
320 u64,
321 recent_coverage_update.items[@sizeOf(abi.fuzz.CoverageUpdateHeader)..][0 .. n_bitset_elems * @sizeOf(u64)],
322 );
323 var sli: SourceLocationIndex = @enumFromInt(0);
324 for (covered_bits) |elem| {
325 try covered.ensureUnusedCapacity(gpa, 64);
326 for (0..@bitSizeOf(u64)) |i| {
327 if ((elem & (@as(u64, 1) << @intCast(i))) != 0) {
328 if (sli.ptr().file == want_file) {
329 covered.appendAssumeCapacity(sli);
330 }
331 }
332 sli = @enumFromInt(@intFromEnum(sli) + 1);
333 }
334 }
335
336 js.updateCoverage(covered.items.ptr, covered.items.len);
337}
338
339fn updateSource() error{OutOfMemory}!void {
340 if (recent_coverage_update.items.len == 0) return;
341 const file_sli = selected_source_location.?;
342 var html: std.ArrayListUnmanaged(u8) = .empty;
343 defer html.deinit(gpa);
344 file_sli.fileHtml(&html) catch |err| switch (err) {
345 error.OutOfMemory => |e| return e,
346 error.SourceUnavailable => {},
347 };
348 js.updateSource(html.items.ptr, html.items.len);
349}
350
351var selected_source_location: ?SourceLocationIndex = null;
352
353/// This function is not used directly by `main.js`, but a reference to it is
354/// emitted by `SourceLocationIndex.sourceLocationLinkHtml`.
355export fn fuzzSelectSli(sli: SourceLocationIndex) void {
356 if (!sli.haveCoverage()) return;
357 selected_source_location = sli;
358 updateEntryPoints() catch @panic("out of memory"); // highlights the selected one green
359 updateSource() catch @panic("out of memory");
360 updateCoverage() catch @panic("out of memory");
361}
362
363const std = @import("std");
364const Allocator = std.mem.Allocator;
365const Coverage = std.debug.Coverage;
366const abi = std.Build.abi;
367const assert = std.debug.assert;
368const gpa = std.heap.wasm_allocator;
369
370const Walk = @import("Walk");
371const html_render = @import("html_render");
372
373const nsSince = @import("main.zig").nsSince;
374const Slice = @import("main.zig").Slice;
375const fatal = @import("main.zig").fatal;
376const log = std.log;
377const String = Slice(u8);
lib/build-web/index.html created+202
......@@ -0,0 +1,202 @@
1<!doctype html>
2
3<meta charset="utf-8">
4<title>Zig Build System</title>
5<link rel="stylesheet" href="style.css">
6<!-- Highly compressed 32x32 Zig logo -->
7<link rel="icon" href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAABSklEQVRYw8WWXbLDIAiFP5xuURYpi+Q+VDvJTYxaY8pLJ52EA5zDj/AD8wRABCw8DeyJBDiAKMiDGaecNYCKYgCvh4EBjPgGh0UVqAB/MEU3D57efDRMiRhWddprCljRAECPCE0Uw4iz4Jn3tP2zFYAB6on4/8NBM1Es+9kl0aKgaMRnwHPpT5MIDb6YzLzp57wNIyIC7iCCdijeL3gv78jZe6cVENn/drRbXbxl4lXSmB3FtbY0iNrjIEwMm6u2VFFjWQCN0qtov6+wANxG/IV7eR8DHw6gzft4NuEXvA8HcDfv31SgyvsMeDUA90/WTd47bsCdv8PUrWzDyw02uIYv13ktgOVr+IqCouila7gWgNYuly/BfVSEdsP5Vdqyiz7pPC40C+p2e21bL5/dByGtAD6eZPuzeznwjoIN748BfyqwmVDyJHCxPwLSkjUkraEXAAAAAElFTkSuQmCC">
8
9<!-- Templates, to be cloned into shadow DOMs by JavaScript -->
10
11<template id="timeReportEntryTemplate">
12 <link rel="stylesheet" href="style.css">
13 <link rel="stylesheet" href="time_report.css">
14 <details>
15 <summary><slot name="step-name"></slot></summary>
16 <div id="genericReport">
17 <div class="stats">
18 Time: <slot name="stat-total-time"></slot><br>
19 </div>
20 </div>
21 <div id="compileReport">
22 <div class="stats">
23 Files Discovered: <slot name="stat-reachable-files"></slot><br>
24 Files Analyzed: <slot name="stat-imported-files"></slot><br>
25 Generic Instances Analyzed: <slot name="stat-generic-instances"></slot><br>
26 Inline Calls Analyzed: <slot name="stat-inline-calls"></slot><br>
27 Compilation Time: <slot name="stat-compilation-time"></slot><br>
28 </div>
29 <table class="time-stats">
30 <thead>
31 <tr>
32 <th scope="col">Pipeline Component</th>
33 <th scope="col" class="tooltip">CPU Time
34 <span class="tooltip-content">Sum across all threads of the time spent in this pipeline component</span>
35 </th>
36 <th scope="col" class="tooltip">Real Time
37 <span class="tooltip-content">Wall-clock time elapsed between the start and end of this compilation phase</span>
38 </th>
39 <th scope="col">Compilation Phase</th>
40 </tr>
41 </thead>
42 <tbody>
43 <tr>
44 <th scope="row" class="tooltip">Parsing
45 <span class="tooltip-content"><code>tokenize</code> converts a file of Zig source code into a sequence of tokens, which are then processed by <code>Parse</code> into an Abstract Syntax Tree (AST).</span>
46 </th>
47 <td><slot name="cpu-time-parse"></slot></td>
48 <td rowspan="2"><slot name="real-time-files"></slot></td>
49 <th scope="row" rowspan="2" class="tooltip">File Lower
50 <span class="tooltip-content">Tokenization, parsing, and lowering of Zig source files to a high-level IR.<br><br>Starting from module roots, every file theoretically accessible through a chain of <code>@import</code> calls is processed. Individual source files are processed serially, but different files are processed in parallel by a thread pool.<br><br>The results of this phase of compilation are cached on disk per source file, meaning the time spent here is typically only relevant to "clean" builds.</span>
51 </th>
52 </tr>
53 <tr>
54 <th scope="row" class="tooltip">AST Lowering
55 <span class="tooltip-content"><code>AstGen</code> converts a file's AST into a high-level SSA IR named Zig Intermediate Representation (ZIR). The resulting ZIR code is cached on disk to avoid, for instance, re-lowering all source files in the Zig standard library each time the compiler is invoked.</span>
56 </th>
57 <td><slot name="cpu-time-astgen"></slot></td>
58 </tr>
59 <tr>
60 <th scope="row" class="tooltip">Semantic Analysis
61 <span class="tooltip-content"><code>Sema</code> interprets ZIR to perform type checking, compile-time code execution, and type resolution, collectively termed "semantic analysis". When a runtime function body is analyzed, it emits Analyzed Intermediate Representation (AIR) code to be sent to the next pipeline component. Semantic analysis is currently entirely single-threaded.</span>
62 </th>
63 <td><slot name="cpu-time-sema"></slot></td>
64 <td rowspan="3"><slot name="real-time-decls"></slot></td>
65 <th scope="row" rowspan="3" class="tooltip">Declaration Lower
66 <span class="tooltip-content">Semantic analysis, code generation, and linking, at the granularity of individual declarations (as opposed to whole source files).<br><br>These components are run in parallel with one another. Semantic analysis is almost always the bottleneck, as it is complex and currently can only run single-threaded.<br><br>This phase completes when a work queue empties, but semantic analysis may add work by one declaration referencing another.<br><br>This is the main phase of compilation, typically taking significantly longer than File Lower (even in a clean build).</span>
67 </th>
68 </tr>
69 <tr>
70 <th scope="row" class="tooltip">Code Generation
71 <span class="tooltip-content"><code>CodeGen</code> converts AIR from <code>Sema</code> into machine instructions in the form of Machine Intermediate Representation (MIR). This work is usually highly parallel, since in most cases, arbitrarily many functions can be run through <code>CodeGen</code> simultaneously.</span>
72 </th>
73 <td><slot name="cpu-time-codegen"></slot></td>
74 </tr>
75 <tr>
76 <th scope="row" class="tooltip">Linking
77 <span class="tooltip-content"><code>link</code> converts MIR from <code>CodeGen</code>, as well as global constants and variables from <code>Sema</code>, and places them in the output binary. MIR is converted to a finished sequence of real instruction bytes.<br><br>When using the LLVM backend, most of this work is instead deferred to the "LLVM Emit" phase.</span>
78 </th>
79 <td><slot name="cpu-time-link"></slot></td>
80 </tr>
81 <tr class="llvm-only">
82 <th class="empty-cell"></th>
83 <td class="empty-cell"></td>
84 <td><slot name="real-time-llvm-emit"></slot></td>
85 <th scope="row" class="tooltip">LLVM Emit
86 <span class="tooltip-content"><b>Only applicable when using the LLVM backend.</b><br><br>Conversion of generated LLVM bitcode to an object file, including any optimization passes.<br><br>When using LLVM, this phase of compilation is typically the slowest by a significant margin. Unfortunately, the Zig compiler implementation has essentially no control over it.</span>
87 </th>
88 </tr>
89 <tr>
90 <th class="empty-cell"></th>
91 <td class="empty-cell"></td>
92 <td><slot name="real-time-link-flush"></slot></td>
93 <th scope="row" class="tooltip">Linker Flush
94 <span class="tooltip-content">Finalizing the emitted binary, and ensuring it is fully written to disk.<br><br>When using LLD, this phase represents the entire linker invocation. Otherwise, the amount of work performed here is dependent on details of Zig's linker implementation for the particular output format, but typically aims to be fairly minimal.</span>
95 </th>
96 </tr>
97 </tbody>
98 </table>
99 <details class="section">
100 <summary>Files</summary>
101 <table class="time-stats">
102 <thead>
103 <tr>
104 <th scope="col">File</th>
105 <th scope="col">Semantic Analysis</th>
106 <th scope="col">Code Generation</th>
107 <th scope="col">Linking</th>
108 </tr>
109 </thead>
110 <!-- HTML does not allow placing a 'slot' inside of a 'tbody' for backwards-compatibility
111 reasons, so we unfortunately must template on the `id` here. -->
112 <tbody id="fileTableBody"></tbody>
113 </table>
114 </details>
115 <details class="section">
116 <summary>Declarations</summary>
117 <table class="time-stats">
118 <thead>
119 <tr>
120 <th scope="col">File</th>
121 <th scope="col">Declaration</th>
122 <th scope="col" class="tooltip">Analysis Count
123 <span class="tooltip-content">The number of times the compiler analyzed some part of this declaration. If this is a function, <code>inline</code> and <code>comptime</code> calls to it are <i>not</i> included here. Typically, this value is approximately equal to the number of instances of a generic declaration.</span>
124 </th>
125 <th scope="col">Semantic Analysis</th>
126 <th scope="col">Code Generation</th>
127 <th scope="col">Linking</th>
128 </tr>
129 </thead>
130 <!-- HTML does not allow placing a 'slot' inside of a 'tbody' for backwards-compatibility
131 reasons, so we unfortunately must template on the `id` here. -->
132 <tbody id="declTableBody"></tbody>
133 </table>
134 </details>
135 <details class="section llvm-only">
136 <summary>LLVM Pass Timings</summary>
137 <div><slot name="llvm-pass-timings"></slot></div>
138 </details>
139 </div>
140 </details>
141</template>
142
143<template id="fuzzEntryTemplate">
144 <link rel="stylesheet" href="style.css">
145 <ul>
146 <li>Total Runs: <slot name="stat-total-runs"></slot></li>
147 <li>Unique Runs: <slot name="stat-unique-runs"></slot></li>
148 <li>Speed: <slot name="stat-speed"></slot> runs/sec</li>
149 <li>Coverage: <slot name="stat-coverage"></slot></li>
150 </ul>
151 <!-- I have observed issues in Firefox clicking frequently-updating slotted links, so the entry
152 point list is handled separately since it rarely changes. -->
153 <ul id="entryPointList" class="no-marker"></ul>
154 <div id="source" class="hidden">
155 <h2>Source Code</h2>
156 <pre><code id="sourceText"></code></pre>
157 </div>
158</template>
159
160<!-- The actual body: fairly minimal, content populated by JavaScript -->
161
162<p id="connectionStatus">Loading JavaScript...</p>
163<p class="hidden" id="firefoxWebSocketBullshitExplainer">
164If you are using Firefox and <code>zig build --listen</code> is definitely running, you may be experiencing an unreasonably aggressive exponential
165backoff for WebSocket connection attempts, which is enabled by default and can block connection attempts for up to a minute. To disable this limit,
166open <code>about:config</code> and set the <code>network.websocket.delay-failed-reconnects</code> option to <code>false</code>.
167</p>
168<main class="hidden">
169 <h1>Zig Build System</h1>
170
171 <p><span id="summaryStatus"></span> | <span id="summaryStepCount"></span> steps</p>
172 <button class="big-btn" id="buttonRebuild" disabled>Rebuild</button>
173
174 <ul class="no-marker" id="stepList"></ul>
175
176 <hr>
177
178 <div id="timeReport" class="hidden">
179 <h1>Time Report</h1>
180 <div id="timeReportList"></div>
181 <hr>
182 </div>
183
184 <div id="fuzz" class="hidden">
185 <h1>Fuzzer</h1>
186 <p id="fuzzStatus"></p>
187 <div id="fuzzEntries"></div>
188 <hr>
189 </div>
190
191 <h1>Help</h1>
192 <p>This is the Zig Build System web interface. It allows live interaction with the build system.</p>
193 <p>The following <code>zig build</code> flags can expose extra features of this interface:</p>
194 <ul>
195 <li><code>--time-report</code>: collect and show statistics about the time taken to evaluate a build graph</li>
196 <li><code>--fuzz</code>: enable the fuzzer for any Zig test binaries in the build graph (experimental)</li>
197 </ul>
198</main>
199
200<!-- JavaScript at the very end -->
201
202<script src="main.js"></script>
lib/build-web/main.js created+346
......@@ -0,0 +1,346 @@
1const domConnectionStatus = document.getElementById("connectionStatus");
2const domFirefoxWebSocketBullshitExplainer = document.getElementById("firefoxWebSocketBullshitExplainer");
3
4const domMain = document.getElementsByTagName("main")[0];
5const domSummary = {
6 stepCount: document.getElementById("summaryStepCount"),
7 status: document.getElementById("summaryStatus"),
8};
9const domButtonRebuild = document.getElementById("buttonRebuild");
10const domStepList = document.getElementById("stepList");
11let domSteps = [];
12
13let wasm_promise = fetch("main.wasm");
14let wasm_exports = null;
15
16const text_decoder = new TextDecoder();
17const text_encoder = new TextEncoder();
18
19domButtonRebuild.addEventListener("click", () => wasm_exports.rebuild());
20
21setConnectionStatus("Loading WebAssembly...", false);
22WebAssembly.instantiateStreaming(wasm_promise, {
23 core: {
24 log: function(ptr, len) {
25 const msg = decodeString(ptr, len);
26 console.log(msg);
27 },
28 panic: function (ptr, len) {
29 const msg = decodeString(ptr, len);
30 throw new Error("panic: " + msg);
31 },
32 timestamp: function () {
33 return BigInt(new Date());
34 },
35 hello: hello,
36 updateBuildStatus: updateBuildStatus,
37 updateStepStatus: updateStepStatus,
38 sendWsMessage: (ptr, len) => ws.send(new Uint8Array(wasm_exports.memory.buffer, ptr, len)),
39 },
40 fuzz: {
41 requestSources: fuzzRequestSources,
42 ready: fuzzReady,
43 updateStats: fuzzUpdateStats,
44 updateEntryPoints: fuzzUpdateEntryPoints,
45 updateSource: fuzzUpdateSource,
46 updateCoverage: fuzzUpdateCoverage,
47 },
48 time_report: {
49 updateCompile: timeReportUpdateCompile,
50 updateGeneric: timeReportUpdateGeneric,
51 },
52}).then(function(obj) {
53 setConnectionStatus("Connecting to WebSocket...", true);
54 connectWebSocket();
55
56 wasm_exports = obj.instance.exports;
57 window.wasm = obj; // for debugging
58});
59
60function connectWebSocket() {
61 const host = document.location.host;
62 const pathname = document.location.pathname;
63 const isHttps = document.location.protocol === 'https:';
64 const match = host.match(/^(.+):(\d+)$/);
65 const defaultPort = isHttps ? 443 : 80;
66 const port = match ? parseInt(match[2], 10) : defaultPort;
67 const hostName = match ? match[1] : host;
68 const wsProto = isHttps ? "wss:" : "ws:";
69 const wsUrl = wsProto + '//' + hostName + ':' + port + pathname;
70 ws = new WebSocket(wsUrl);
71 ws.binaryType = "arraybuffer";
72 ws.addEventListener('message', onWebSocketMessage, false);
73 ws.addEventListener('error', onWebSocketClose, false);
74 ws.addEventListener('close', onWebSocketClose, false);
75 ws.addEventListener('open', onWebSocketOpen, false);
76}
77function onWebSocketOpen() {
78 setConnectionStatus("Waiting for data...", false);
79}
80function onWebSocketMessage(ev) {
81 const jsArray = new Uint8Array(ev.data);
82 const ptr = wasm_exports.message_begin(jsArray.length);
83 const wasmArray = new Uint8Array(wasm_exports.memory.buffer, ptr, jsArray.length);
84 wasmArray.set(jsArray);
85 wasm_exports.message_end();
86}
87function onWebSocketClose() {
88 setConnectionStatus("WebSocket connection closed. Re-connecting...", true);
89 ws.removeEventListener('message', onWebSocketMessage, false);
90 ws.removeEventListener('error', onWebSocketClose, false);
91 ws.removeEventListener('close', onWebSocketClose, false);
92 ws.removeEventListener('open', onWebSocketOpen, false);
93 ws = null;
94 setTimeout(connectWebSocket, 1000);
95}
96
97function setConnectionStatus(msg, is_websocket_connect) {
98 domConnectionStatus.textContent = msg;
99 if (msg.length > 0) {
100 domConnectionStatus.classList.remove("hidden");
101 domMain.classList.add("hidden");
102 } else {
103 domConnectionStatus.classList.add("hidden");
104 domMain.classList.remove("hidden");
105 }
106 if (is_websocket_connect) {
107 domFirefoxWebSocketBullshitExplainer.classList.remove("hidden");
108 } else {
109 domFirefoxWebSocketBullshitExplainer.classList.add("hidden");
110 }
111}
112
113function hello(
114 steps_len,
115 build_status,
116 time_report,
117) {
118 domSummary.stepCount.textContent = steps_len;
119 updateBuildStatus(build_status);
120 setConnectionStatus("", false);
121
122 {
123 let entries = [];
124 for (let i = 0; i < steps_len; i += 1) {
125 const step_name = unwrapString(wasm_exports.stepName(i));
126 const code = document.createElement("code");
127 code.textContent = step_name;
128 const li = document.createElement("li");
129 li.appendChild(code);
130 entries.push(li);
131 }
132 domStepList.replaceChildren(...entries);
133 for (let i = 0; i < steps_len; i += 1) {
134 updateStepStatus(i);
135 }
136 }
137
138 if (time_report) timeReportReset(steps_len);
139 fuzzReset();
140}
141
142function updateBuildStatus(s) {
143 let text;
144 let active = false;
145 let reset_time_reports = false;
146 if (s == 0) {
147 text = "Idle";
148 } else if (s == 1) {
149 text = "Watching for changes...";
150 } else if (s == 2) {
151 text = "Running...";
152 active = true;
153 reset_time_reports = true;
154 } else if (s == 3) {
155 text = "Starting fuzzer...";
156 active = true;
157 } else {
158 console.log(`bad build status: ${s}`);
159 }
160 domSummary.status.textContent = text;
161 if (active) {
162 domSummary.status.classList.add("status-running");
163 domSummary.status.classList.remove("status-idle");
164 domButtonRebuild.disabled = true;
165 } else {
166 domSummary.status.classList.remove("status-running");
167 domSummary.status.classList.add("status-idle");
168 domButtonRebuild.disabled = false;
169 }
170 if (reset_time_reports) {
171 // Grey out and collapse all the time reports
172 for (const time_report_host of domTimeReportList.children) {
173 const details = time_report_host.shadowRoot.querySelector(":host > details");
174 details.classList.add("pending");
175 details.open = false;
176 }
177 }
178}
179function updateStepStatus(step_idx) {
180 const li = domStepList.children[step_idx];
181 const step_status = wasm_exports.stepStatus(step_idx);
182 li.classList.remove("step-wip", "step-success", "step-failure");
183 if (step_status == 0) {
184 // pending
185 } else if (step_status == 1) {
186 li.classList.add("step-wip");
187 } else if (step_status == 2) {
188 li.classList.add("step-success");
189 } else if (step_status == 3) {
190 li.classList.add("step-failure");
191 } else {
192 console.log(`bad step status: ${step_status}`);
193 }
194}
195
196function decodeString(ptr, len) {
197 if (len === 0) return "";
198 return text_decoder.decode(new Uint8Array(wasm_exports.memory.buffer, ptr, len));
199}
200function getU32Array(ptr, len) {
201 if (len === 0) return new Uint32Array();
202 return new Uint32Array(wasm_exports.memory.buffer, ptr, len);
203}
204function unwrapString(bigint) {
205 const ptr = Number(bigint & 0xffffffffn);
206 const len = Number(bigint >> 32n);
207 return decodeString(ptr, len);
208}
209
210const time_report_entry_template = document.getElementById("timeReportEntryTemplate").content;
211const domTimeReport = document.getElementById("timeReport");
212const domTimeReportList = document.getElementById("timeReportList");
213function timeReportReset(steps_len) {
214 let entries = [];
215 for (let i = 0; i < steps_len; i += 1) {
216 const step_name = unwrapString(wasm_exports.stepName(i));
217 const host = document.createElement("div");
218 const shadow = host.attachShadow({ mode: "open" });
219 shadow.appendChild(time_report_entry_template.cloneNode(true));
220 shadow.querySelector(":host > details").classList.add("pending");
221 const slotted_name = document.createElement("code");
222 slotted_name.setAttribute("slot", "step-name");
223 slotted_name.textContent = step_name;
224 host.appendChild(slotted_name);
225 entries.push(host);
226 }
227 domTimeReportList.replaceChildren(...entries);
228 domTimeReport.classList.remove("hidden");
229}
230function timeReportUpdateCompile(
231 step_idx,
232 inner_html_ptr,
233 inner_html_len,
234 file_table_html_ptr,
235 file_table_html_len,
236 decl_table_html_ptr,
237 decl_table_html_len,
238 use_llvm,
239) {
240 const inner_html = decodeString(inner_html_ptr, inner_html_len);
241 const file_table_html = decodeString(file_table_html_ptr, file_table_html_len);
242 const decl_table_html = decodeString(decl_table_html_ptr, decl_table_html_len);
243
244 const host = domTimeReportList.children.item(step_idx);
245 const shadow = host.shadowRoot;
246
247 shadow.querySelector(":host > details").classList.remove("pending", "no-llvm");
248
249 shadow.getElementById("genericReport").classList.add("hidden");
250 shadow.getElementById("compileReport").classList.remove("hidden");
251
252 if (!use_llvm) shadow.querySelector(":host > details").classList.add("no-llvm");
253 host.innerHTML = inner_html;
254 shadow.getElementById("fileTableBody").innerHTML = file_table_html;
255 shadow.getElementById("declTableBody").innerHTML = decl_table_html;
256}
257function timeReportUpdateGeneric(
258 step_idx,
259 inner_html_ptr,
260 inner_html_len,
261) {
262 const inner_html = decodeString(inner_html_ptr, inner_html_len);
263 const host = domTimeReportList.children.item(step_idx);
264 const shadow = host.shadowRoot;
265 shadow.querySelector(":host > details").classList.remove("pending", "no-llvm");
266 shadow.getElementById("genericReport").classList.remove("hidden");
267 shadow.getElementById("compileReport").classList.add("hidden");
268 host.innerHTML = inner_html;
269}
270
271const fuzz_entry_template = document.getElementById("fuzzEntryTemplate").content;
272const domFuzz = document.getElementById("fuzz");
273const domFuzzStatus = document.getElementById("fuzzStatus");
274const domFuzzEntries = document.getElementById("fuzzEntries");
275let domFuzzInstance = null;
276function fuzzRequestSources() {
277 domFuzzStatus.classList.remove("hidden");
278 domFuzzStatus.textContent = "Loading sources tarball...";
279 fetch("sources.tar").then(function(response) {
280 if (!response.ok) throw new Error("unable to download sources");
281 domFuzzStatus.textContent = "Parsing fuzz test sources...";
282 return response.arrayBuffer();
283 }).then(function(buffer) {
284 if (buffer.length === 0) throw new Error("sources.tar was empty");
285 const js_array = new Uint8Array(buffer);
286 const ptr = wasm_exports.alloc(js_array.length);
287 const wasm_array = new Uint8Array(wasm_exports.memory.buffer, ptr, js_array.length);
288 wasm_array.set(js_array);
289 wasm_exports.fuzzUnpackSources(ptr, js_array.length);
290 domFuzzStatus.textContent = "";
291 domFuzzStatus.classList.add("hidden");
292 });
293}
294function fuzzReady() {
295 domFuzz.classList.remove("hidden");
296
297 // TODO: multiple fuzzer instances
298 if (domFuzzInstance !== null) return;
299
300 const host = document.createElement("div");
301 const shadow = host.attachShadow({ mode: "open" });
302 shadow.appendChild(fuzz_entry_template.cloneNode(true));
303
304 domFuzzInstance = host;
305 domFuzzEntries.appendChild(host);
306}
307function fuzzReset() {
308 domFuzz.classList.add("hidden");
309 domFuzzEntries.replaceChildren();
310 domFuzzInstance = null;
311}
312function fuzzUpdateStats(stats_html_ptr, stats_html_len) {
313 if (domFuzzInstance === null) throw new Error("fuzzUpdateStats called when fuzzer inactive");
314 const stats_html = decodeString(stats_html_ptr, stats_html_len);
315 const host = domFuzzInstance;
316 host.innerHTML = stats_html;
317}
318function fuzzUpdateEntryPoints(entry_points_html_ptr, entry_points_html_len) {
319 if (domFuzzInstance === null) throw new Error("fuzzUpdateEntryPoints called when fuzzer inactive");
320 const entry_points_html = decodeString(entry_points_html_ptr, entry_points_html_len);
321 const domEntryPointList = domFuzzInstance.shadowRoot.getElementById("entryPointList");
322 domEntryPointList.innerHTML = entry_points_html;
323}
324function fuzzUpdateSource(source_html_ptr, source_html_len) {
325 if (domFuzzInstance === null) throw new Error("fuzzUpdateSource called when fuzzer inactive");
326 const source_html = decodeString(source_html_ptr, source_html_len);
327 const domSourceText = domFuzzInstance.shadowRoot.getElementById("sourceText");
328 domSourceText.innerHTML = source_html;
329 domFuzzInstance.shadowRoot.getElementById("source").classList.remove("hidden");
330}
331function fuzzUpdateCoverage(covered_ptr, covered_len) {
332 if (domFuzzInstance === null) throw new Error("fuzzUpdateCoverage called when fuzzer inactive");
333 const shadow = domFuzzInstance.shadowRoot;
334 const domSourceText = shadow.getElementById("sourceText");
335 const covered = getU32Array(covered_ptr, covered_len);
336 for (let i = 0; i < domSourceText.children.length; i += 1) {
337 const childDom = domSourceText.children[i];
338 if (childDom.id != null && childDom.id[0] == "l") {
339 childDom.classList.add("l");
340 childDom.classList.remove("c");
341 }
342 }
343 for (const sli of covered) {
344 shadow.getElementById(`l${sli}`).classList.add("c");
345 }
346}
lib/build-web/main.zig created+213
......@@ -0,0 +1,213 @@
1const std = @import("std");
2const assert = std.debug.assert;
3const abi = std.Build.abi;
4const gpa = std.heap.wasm_allocator;
5const log = std.log;
6const Allocator = std.mem.Allocator;
7
8const fuzz = @import("fuzz.zig");
9const time_report = @import("time_report.zig");
10
11/// Nanoseconds.
12var server_base_timestamp: i64 = 0;
13/// Milliseconds.
14var client_base_timestamp: i64 = 0;
15
16pub var step_list: []Step = &.{};
17/// Not accessed after initialization, but must be freed alongside `step_list`.
18pub var step_list_data: []u8 = &.{};
19
20const Step = struct {
21 name: []const u8,
22 status: abi.StepUpdate.Status,
23};
24
25const js = struct {
26 extern "core" fn log(ptr: [*]const u8, len: usize) void;
27 extern "core" fn panic(ptr: [*]const u8, len: usize) noreturn;
28 extern "core" fn timestamp() i64;
29 extern "core" fn hello(
30 steps_len: u32,
31 status: abi.BuildStatus,
32 time_report: bool,
33 ) void;
34 extern "core" fn updateBuildStatus(status: abi.BuildStatus) void;
35 extern "core" fn updateStepStatus(step_idx: u32) void;
36 extern "core" fn sendWsMessage(ptr: [*]const u8, len: usize) void;
37};
38
39pub const std_options: std.Options = .{
40 .logFn = logFn,
41};
42
43pub fn panic(msg: []const u8, st: ?*std.builtin.StackTrace, addr: ?usize) noreturn {
44 _ = st;
45 _ = addr;
46 log.err("panic: {s}", .{msg});
47 @trap();
48}
49
50fn logFn(
51 comptime message_level: log.Level,
52 comptime scope: @TypeOf(.enum_literal),
53 comptime format: []const u8,
54 args: anytype,
55) void {
56 const level_txt = comptime message_level.asText();
57 const prefix2 = if (scope == .default) ": " else "(" ++ @tagName(scope) ++ "): ";
58 var buf: [500]u8 = undefined;
59 const line = std.fmt.bufPrint(&buf, level_txt ++ prefix2 ++ format, args) catch l: {
60 buf[buf.len - 3 ..][0..3].* = "...".*;
61 break :l &buf;
62 };
63 js.log(line.ptr, line.len);
64}
65
66export fn alloc(n: usize) [*]u8 {
67 const slice = gpa.alloc(u8, n) catch @panic("OOM");
68 return slice.ptr;
69}
70
71var message_buffer: std.ArrayListAlignedUnmanaged(u8, .of(u64)) = .empty;
72
73/// Resizes the message buffer to be the correct length; returns the pointer to
74/// the query string.
75export fn message_begin(len: usize) [*]u8 {
76 message_buffer.resize(gpa, len) catch @panic("OOM");
77 return message_buffer.items.ptr;
78}
79
80export fn message_end() void {
81 const msg_bytes = message_buffer.items;
82
83 const tag: abi.ToClientTag = @enumFromInt(msg_bytes[0]);
84 switch (tag) {
85 _ => @panic("malformed message"),
86
87 .hello => return helloMessage(msg_bytes) catch @panic("OOM"),
88 .status_update => return statusUpdateMessage(msg_bytes) catch @panic("OOM"),
89 .step_update => return stepUpdateMessage(msg_bytes) catch @panic("OOM"),
90
91 .fuzz_source_index => return fuzz.sourceIndexMessage(msg_bytes) catch @panic("OOM"),
92 .fuzz_coverage_update => return fuzz.coverageUpdateMessage(msg_bytes) catch @panic("OOM"),
93 .fuzz_entry_points => return fuzz.entryPointsMessage(msg_bytes) catch @panic("OOM"),
94
95 .time_report_generic_result => return time_report.genericResultMessage(msg_bytes) catch @panic("OOM"),
96 .time_report_compile_result => return time_report.compileResultMessage(msg_bytes) catch @panic("OOM"),
97 }
98}
99
100const String = Slice(u8);
101
102pub fn Slice(T: type) type {
103 return packed struct(u64) {
104 ptr: u32,
105 len: u32,
106
107 pub fn init(s: []const T) @This() {
108 return .{
109 .ptr = @intFromPtr(s.ptr),
110 .len = s.len,
111 };
112 }
113 };
114}
115
116pub fn fatal(comptime format: []const u8, args: anytype) noreturn {
117 var buf: [500]u8 = undefined;
118 const line = std.fmt.bufPrint(&buf, format, args) catch l: {
119 buf[buf.len - 3 ..][0..3].* = "...".*;
120 break :l &buf;
121 };
122 js.panic(line.ptr, line.len);
123}
124
125fn helloMessage(msg_bytes: []align(4) u8) Allocator.Error!void {
126 if (msg_bytes.len < @sizeOf(abi.Hello)) @panic("malformed Hello message");
127 const hdr: *const abi.Hello = @ptrCast(msg_bytes[0..@sizeOf(abi.Hello)]);
128 const trailing = msg_bytes[@sizeOf(abi.Hello)..];
129
130 client_base_timestamp = js.timestamp();
131 server_base_timestamp = hdr.timestamp;
132
133 const steps = try gpa.alloc(Step, hdr.steps_len);
134 errdefer gpa.free(steps);
135
136 const step_name_lens: []align(1) const u32 = @ptrCast(trailing[0 .. steps.len * 4]);
137
138 const step_name_data_len: usize = len: {
139 var sum: usize = 0;
140 for (step_name_lens) |n| sum += n;
141 break :len sum;
142 };
143 const step_name_data: []const u8 = trailing[steps.len * 4 ..][0..step_name_data_len];
144 const step_status_bits: []const u8 = trailing[steps.len * 4 + step_name_data_len ..];
145
146 const duped_step_name_data = try gpa.dupe(u8, step_name_data);
147 errdefer gpa.free(duped_step_name_data);
148
149 var name_off: usize = 0;
150 for (steps, step_name_lens, 0..) |*step_out, name_len, step_idx| {
151 step_out.* = .{
152 .name = duped_step_name_data[name_off..][0..name_len],
153 .status = @enumFromInt(@as(u2, @truncate(step_status_bits[step_idx / 4] >> @intCast((step_idx % 4) * 2)))),
154 };
155 name_off += name_len;
156 }
157
158 gpa.free(step_list);
159 gpa.free(step_list_data);
160 step_list = steps;
161 step_list_data = duped_step_name_data;
162
163 js.hello(step_list.len, hdr.status, hdr.flags.time_report);
164}
165fn statusUpdateMessage(msg_bytes: []u8) Allocator.Error!void {
166 if (msg_bytes.len < @sizeOf(abi.StatusUpdate)) @panic("malformed StatusUpdate message");
167 const msg: *const abi.StatusUpdate = @ptrCast(msg_bytes[0..@sizeOf(abi.StatusUpdate)]);
168 js.updateBuildStatus(msg.new);
169}
170fn stepUpdateMessage(msg_bytes: []u8) Allocator.Error!void {
171 if (msg_bytes.len < @sizeOf(abi.StepUpdate)) @panic("malformed StepUpdate message");
172 const msg: *const abi.StepUpdate = @ptrCast(msg_bytes[0..@sizeOf(abi.StepUpdate)]);
173 if (msg.step_idx >= step_list.len) @panic("malformed StepUpdate message");
174 step_list[msg.step_idx].status = msg.bits.status;
175 js.updateStepStatus(msg.step_idx);
176}
177
178export fn stepName(idx: usize) String {
179 return .init(step_list[idx].name);
180}
181export fn stepStatus(idx: usize) u8 {
182 return @intFromEnum(step_list[idx].status);
183}
184
185export fn rebuild() void {
186 const msg: abi.Rebuild = .{};
187 const raw: []const u8 = @ptrCast(&msg);
188 js.sendWsMessage(raw.ptr, raw.len);
189}
190
191/// Nanoseconds passed since a server timestamp.
192pub fn nsSince(server_timestamp: i64) i64 {
193 const ms_passed = js.timestamp() - client_base_timestamp;
194 const ns_passed = server_base_timestamp - server_timestamp;
195 return ns_passed + ms_passed * std.time.ns_per_ms;
196}
197
198pub fn fmtEscapeHtml(unescaped: []const u8) HtmlEscaper {
199 return .{ .unescaped = unescaped };
200}
201const HtmlEscaper = struct {
202 unescaped: []const u8,
203 pub fn format(he: HtmlEscaper, w: *std.Io.Writer) !void {
204 for (he.unescaped) |c| switch (c) {
205 '&' => try w.writeAll("&amp;"),
206 '<' => try w.writeAll("&lt;"),
207 '>' => try w.writeAll("&gt;"),
208 '"' => try w.writeAll("&quot;"),
209 '\'' => try w.writeAll("&#39;"),
210 else => try w.writeByte(c),
211 };
212 }
213};
lib/build-web/style.css created+240
......@@ -0,0 +1,240 @@
1body {
2 font-family: system-ui, -apple-system, Roboto, "Segoe UI", sans-serif;
3 color: #000000;
4 padding: 1em 10%;
5}
6ul.no-marker {
7 list-style-type: none;
8 padding-left: 0;
9}
10hr {
11 margin: 2em 0;
12}
13.hidden {
14 display: none;
15}
16.empty-cell {
17 background: #ccc;
18}
19table.time-stats > tbody > tr > th {
20 text-align: left;
21}
22table.time-stats > tbody > tr > td {
23 text-align: right;
24}
25details > summary {
26 cursor: pointer;
27 font-size: 1.5em;
28}
29.tooltip {
30 text-decoration: underline;
31 cursor: help;
32}
33.tooltip-content {
34 border-radius: 6px;
35 display: none;
36 position: absolute;
37 background: #fff;
38 border: 1px solid black;
39 max-width: 500px;
40 padding: 1em;
41 text-align: left;
42 font-weight: normal;
43 pointer-events: none;
44}
45.tooltip:hover > .tooltip-content {
46 display: block;
47}
48table {
49 margin: 1.0em auto 1.5em 0;
50 border-collapse: collapse;
51}
52th, td {
53 padding: 0.5em 1em 0.5em 1em;
54 border: 1px solid;
55 border-color: black;
56}
57a, button {
58 color: #2A6286;
59}
60button {
61 background: #eee;
62 cursor: pointer;
63 border: none;
64 border-radius: 3px;
65 padding: 0.2em 0.5em;
66}
67button.big-btn {
68 font-size: 1.3em;
69}
70button.linkish {
71 background: none;
72 text-decoration: underline;
73 padding: 0;
74}
75button:disabled {
76 color: #888;
77 cursor: not-allowed;
78}
79pre {
80 font-family: "Source Code Pro", monospace;
81 font-size: 1em;
82 background-color: #F5F5F5;
83 padding: 1em;
84 margin: 0;
85 overflow-x: auto;
86}
87:not(pre) > code {
88 white-space: break-spaces;
89}
90code {
91 font-family: "Source Code Pro", monospace;
92 font-size: 0.9em;
93}
94code a {
95 color: #000000;
96}
97kbd {
98 color: #000;
99 background-color: #fafbfc;
100 border-color: #d1d5da;
101 border-bottom-color: #c6cbd1;
102 box-shadow-color: #c6cbd1;
103 display: inline-block;
104 padding: 0.3em 0.2em;
105 font: 1.2em monospace;
106 line-height: 0.8em;
107 vertical-align: middle;
108 border: solid 1px;
109 border-radius: 3px;
110 box-shadow: inset 0 -1px 0;
111 cursor: default;
112}
113.status-running { color: #181; }
114.status-idle { color: #444; }
115.step-success { color: #181; }
116.step-failure { color: #d11; }
117.step-wip::before {
118 content: '';
119 position: absolute;
120 margin-left: -1.5em;
121 width: 1em;
122 text-align: center;
123 animation-name: spinner;
124 animation-duration: 0.5s;
125 animation-iteration-count: infinite;
126 animation-timing-function: step-start;
127}
128@keyframes spinner {
129 0% { content: '|'; }
130 25% { content: '/'; }
131 50% { content: '-'; }
132 75% { content: '\\'; }
133 100% { content: '|'; }
134}
135
136.l {
137 display: inline-block;
138 background: red;
139 width: 1em;
140 height: 1em;
141 border-radius: 1em;
142}
143.c {
144 background-color: green;
145}
146
147.tok-kw {
148 color: #333;
149 font-weight: bold;
150}
151.tok-str {
152 color: #d14;
153}
154.tok-builtin {
155 color: #0086b3;
156}
157.tok-comment {
158 color: #777;
159 font-style: italic;
160}
161.tok-fn {
162 color: #900;
163 font-weight: bold;
164}
165.tok-null {
166 color: #008080;
167}
168.tok-number {
169 color: #008080;
170}
171.tok-type {
172 color: #458;
173 font-weight: bold;
174}
175
176@media (prefers-color-scheme: dark) {
177 body {
178 background-color: #111;
179 color: #ddd;
180 }
181 pre {
182 background-color: #222;
183 }
184 a, button {
185 color: #88f;
186 }
187 button {
188 background: #333;
189 }
190 button:disabled {
191 color: #555;
192 }
193 code a {
194 color: #eee;
195 }
196 th, td {
197 border-color: white;
198 }
199 .empty-cell {
200 background: #000;
201 }
202 .tooltip-content {
203 background: #060606;
204 border-color: white;
205 }
206 .status-running { color: #90ee90; }
207 .status-idle { color: #bbb; }
208 .step-success { color: #90ee90; }
209 .step-failure { color: #f66; }
210 .l {
211 background-color: red;
212 }
213 .c {
214 background-color: green;
215 }
216 .tok-kw {
217 color: #eee;
218 }
219 .tok-str {
220 color: #2e5;
221 }
222 .tok-builtin {
223 color: #ff894c;
224 }
225 .tok-comment {
226 color: #aa7;
227 }
228 .tok-fn {
229 color: #B1A0F8;
230 }
231 .tok-null {
232 color: #ff8080;
233 }
234 .tok-number {
235 color: #ff8080;
236 }
237 .tok-type {
238 color: #68f;
239 }
240}
lib/build-web/time_report.css created+43
......@@ -0,0 +1,43 @@
1:host > details {
2 padding: 0.5em 1em;
3 background: #f2f2f2;
4 margin-bottom: 1.0em;
5 overflow-x: scroll;
6}
7:host > details.pending {
8 pointer-events: none;
9 background: #fafafa;
10 color: #666;
11}
12:host > details > div {
13 margin: 1em 2em;
14 overflow: scroll; /* we'll try to avoid overflow, but if it does happen, this makes sense */
15}
16.stats {
17 font-size: 1.2em;
18}
19details.section {
20 margin: 1.0em 0 0 0;
21}
22details.section > summary {
23 font-weight: bold;
24}
25details.section > :not(summary) {
26 margin-left: 2em;
27}
28:host > details.no-llvm .llvm-only {
29 display: none;
30}
31@media (prefers-color-scheme: dark) {
32 :host > details {
33 background: #222;
34 }
35 :host > details.pending {
36 background: #181818;
37 color: #888;
38 }
39}
40th {
41 max-width: 20em; /* don't let the 'file' column get crazy long */
42 overflow-wrap: anywhere; /* avoid overflow where possible */
43}
lib/build-web/time_report.zig created+234
......@@ -0,0 +1,234 @@
1const std = @import("std");
2const gpa = std.heap.wasm_allocator;
3const abi = std.Build.abi.time_report;
4const fmtEscapeHtml = @import("root").fmtEscapeHtml;
5const step_list = &@import("root").step_list;
6
7const js = struct {
8 extern "time_report" fn updateGeneric(
9 /// The index of the step.
10 step_idx: u32,
11 // The HTML which will be used to populate the template slots.
12 inner_html_ptr: [*]const u8,
13 inner_html_len: usize,
14 ) void;
15 extern "time_report" fn updateCompile(
16 /// The index of the step.
17 step_idx: u32,
18 // The HTML which will be used to populate the template slots.
19 inner_html_ptr: [*]const u8,
20 inner_html_len: usize,
21 // The HTML which will populate the <tbody> of the file table.
22 file_table_html_ptr: [*]const u8,
23 file_table_html_len: usize,
24 // The HTML which will populate the <tbody> of the decl table.
25 decl_table_html_ptr: [*]const u8,
26 decl_table_html_len: usize,
27 /// Whether the LLVM backend was used. If not, LLVM-specific statistics are hidden.
28 use_llvm: bool,
29 ) void;
30};
31
32pub fn genericResultMessage(msg_bytes: []u8) error{OutOfMemory}!void {
33 if (msg_bytes.len != @sizeOf(abi.GenericResult)) @panic("malformed GenericResult message");
34 const msg: *const abi.GenericResult = @ptrCast(msg_bytes);
35 if (msg.step_idx >= step_list.*.len) @panic("malformed GenericResult message");
36 const inner_html = try std.fmt.allocPrint(gpa,
37 \\<code slot="step-name">{[step_name]f}</code>
38 \\<span slot="stat-total-time">{[stat_total_time]D}</span>
39 , .{
40 .step_name = fmtEscapeHtml(step_list.*[msg.step_idx].name),
41 .stat_total_time = msg.ns_total,
42 });
43 defer gpa.free(inner_html);
44 js.updateGeneric(msg.step_idx, inner_html.ptr, inner_html.len);
45}
46
47pub fn compileResultMessage(msg_bytes: []u8) error{OutOfMemory}!void {
48 const max_table_rows = 500;
49
50 if (msg_bytes.len < @sizeOf(abi.CompileResult)) @panic("malformed CompileResult message");
51 const hdr: *const abi.CompileResult = @ptrCast(msg_bytes[0..@sizeOf(abi.CompileResult)]);
52 if (hdr.step_idx >= step_list.*.len) @panic("malformed CompileResult message");
53 var trailing = msg_bytes[@sizeOf(abi.CompileResult)..];
54
55 const llvm_pass_timings = trailing[0..hdr.llvm_pass_timings_len];
56 trailing = trailing[hdr.llvm_pass_timings_len..];
57
58 const FileTimeReport = struct {
59 name: []const u8,
60 ns_sema: u64,
61 ns_codegen: u64,
62 ns_link: u64,
63 };
64 const DeclTimeReport = struct {
65 file_name: []const u8,
66 name: []const u8,
67 sema_count: u32,
68 ns_sema: u64,
69 ns_codegen: u64,
70 ns_link: u64,
71 };
72
73 const slowest_files = try gpa.alloc(FileTimeReport, hdr.files_len);
74 defer gpa.free(slowest_files);
75
76 const slowest_decls = try gpa.alloc(DeclTimeReport, hdr.decls_len);
77 defer gpa.free(slowest_decls);
78
79 for (slowest_files) |*file_out| {
80 const i = std.mem.indexOfScalar(u8, trailing, 0) orelse @panic("malformed CompileResult message");
81 file_out.* = .{
82 .name = trailing[0..i],
83 .ns_sema = 0,
84 .ns_codegen = 0,
85 .ns_link = 0,
86 };
87 trailing = trailing[i + 1 ..];
88 }
89
90 for (slowest_decls) |*decl_out| {
91 const i = std.mem.indexOfScalar(u8, trailing, 0) orelse @panic("malformed CompileResult message");
92 const file_idx = std.mem.readInt(u32, trailing[i..][1..5], .little);
93 const sema_count = std.mem.readInt(u32, trailing[i..][5..9], .little);
94 const sema_ns = std.mem.readInt(u64, trailing[i..][9..17], .little);
95 const codegen_ns = std.mem.readInt(u64, trailing[i..][17..25], .little);
96 const link_ns = std.mem.readInt(u64, trailing[i..][25..33], .little);
97 const file = &slowest_files[file_idx];
98 decl_out.* = .{
99 .file_name = file.name,
100 .name = trailing[0..i],
101 .sema_count = sema_count,
102 .ns_sema = sema_ns,
103 .ns_codegen = codegen_ns,
104 .ns_link = link_ns,
105 };
106 trailing = trailing[i + 33 ..];
107 file.ns_sema += sema_ns;
108 file.ns_codegen += codegen_ns;
109 file.ns_link += link_ns;
110 }
111
112 const S = struct {
113 fn fileLessThan(_: void, lhs: FileTimeReport, rhs: FileTimeReport) bool {
114 const lhs_ns = lhs.ns_sema + lhs.ns_codegen + lhs.ns_link;
115 const rhs_ns = rhs.ns_sema + rhs.ns_codegen + rhs.ns_link;
116 return lhs_ns > rhs_ns; // flipped to sort in reverse order
117 }
118 fn declLessThan(_: void, lhs: DeclTimeReport, rhs: DeclTimeReport) bool {
119 //if (true) return lhs.sema_count > rhs.sema_count;
120 const lhs_ns = lhs.ns_sema + lhs.ns_codegen + lhs.ns_link;
121 const rhs_ns = rhs.ns_sema + rhs.ns_codegen + rhs.ns_link;
122 return lhs_ns > rhs_ns; // flipped to sort in reverse order
123 }
124 };
125 std.mem.sort(FileTimeReport, slowest_files, {}, S.fileLessThan);
126 std.mem.sort(DeclTimeReport, slowest_decls, {}, S.declLessThan);
127
128 const stats = hdr.stats;
129 const inner_html = try std.fmt.allocPrint(gpa,
130 \\<code slot="step-name">{[step_name]f}</code>
131 \\<span slot="stat-reachable-files">{[stat_reachable_files]d}</span>
132 \\<span slot="stat-imported-files">{[stat_imported_files]d}</span>
133 \\<span slot="stat-generic-instances">{[stat_generic_instances]d}</span>
134 \\<span slot="stat-inline-calls">{[stat_inline_calls]d}</span>
135 \\<span slot="stat-compilation-time">{[stat_compilation_time]D}</span>
136 \\<span slot="cpu-time-parse">{[cpu_time_parse]D}</span>
137 \\<span slot="cpu-time-astgen">{[cpu_time_astgen]D}</span>
138 \\<span slot="cpu-time-sema">{[cpu_time_sema]D}</span>
139 \\<span slot="cpu-time-codegen">{[cpu_time_codegen]D}</span>
140 \\<span slot="cpu-time-link">{[cpu_time_link]D}</span>
141 \\<span slot="real-time-files">{[real_time_files]D}</span>
142 \\<span slot="real-time-decls">{[real_time_decls]D}</span>
143 \\<span slot="real-time-llvm-emit">{[real_time_llvm_emit]D}</span>
144 \\<span slot="real-time-link-flush">{[real_time_link_flush]D}</span>
145 \\<pre slot="llvm-pass-timings"><code>{[llvm_pass_timings]f}</code></pre>
146 \\
147 , .{
148 .step_name = fmtEscapeHtml(step_list.*[hdr.step_idx].name),
149 .stat_reachable_files = stats.n_reachable_files,
150 .stat_imported_files = stats.n_imported_files,
151 .stat_generic_instances = stats.n_generic_instances,
152 .stat_inline_calls = stats.n_inline_calls,
153 .stat_compilation_time = hdr.ns_total,
154
155 .cpu_time_parse = stats.cpu_ns_parse,
156 .cpu_time_astgen = stats.cpu_ns_astgen,
157 .cpu_time_sema = stats.cpu_ns_sema,
158 .cpu_time_codegen = stats.cpu_ns_codegen,
159 .cpu_time_link = stats.cpu_ns_link,
160 .real_time_files = stats.real_ns_files,
161 .real_time_decls = stats.real_ns_decls,
162 .real_time_llvm_emit = stats.real_ns_llvm_emit,
163 .real_time_link_flush = stats.real_ns_link_flush,
164
165 .llvm_pass_timings = fmtEscapeHtml(llvm_pass_timings),
166 });
167 defer gpa.free(inner_html);
168
169 var file_table_html: std.ArrayListUnmanaged(u8) = .empty;
170 defer file_table_html.deinit(gpa);
171 for (slowest_files[0..@min(max_table_rows, slowest_files.len)]) |file| {
172 try file_table_html.writer(gpa).print(
173 \\<tr>
174 \\ <th scope="row"><code>{f}</code></th>
175 \\ <td>{D}</td>
176 \\ <td>{D}</td>
177 \\ <td>{D}</td>
178 \\</tr>
179 \\
180 , .{
181 fmtEscapeHtml(file.name),
182 file.ns_sema,
183 file.ns_codegen,
184 file.ns_link,
185 });
186 }
187 if (slowest_files.len > max_table_rows) {
188 try file_table_html.writer(gpa).print(
189 \\<tr><td colspan="4">{d} more rows omitted</td></tr>
190 \\
191 , .{slowest_files.len - max_table_rows});
192 }
193
194 var decl_table_html: std.ArrayListUnmanaged(u8) = .empty;
195 defer decl_table_html.deinit(gpa);
196
197 for (slowest_decls[0..@min(max_table_rows, slowest_decls.len)]) |decl| {
198 try decl_table_html.writer(gpa).print(
199 \\<tr>
200 \\ <th scope="row"><code>{f}</code></th>
201 \\ <th scope="row"><code>{f}</code></th>
202 \\ <td>{d}</td>
203 \\ <td>{D}</td>
204 \\ <td>{D}</td>
205 \\ <td>{D}</td>
206 \\</tr>
207 \\
208 , .{
209 fmtEscapeHtml(decl.file_name),
210 fmtEscapeHtml(decl.name),
211 decl.sema_count,
212 decl.ns_sema,
213 decl.ns_codegen,
214 decl.ns_link,
215 });
216 }
217 if (slowest_decls.len > max_table_rows) {
218 try decl_table_html.writer(gpa).print(
219 \\<tr><td colspan="6">{d} more rows omitted</td></tr>
220 \\
221 , .{slowest_decls.len - max_table_rows});
222 }
223
224 js.updateCompile(
225 hdr.step_idx,
226 inner_html.ptr,
227 inner_html.len,
228 file_table_html.items.ptr,
229 file_table_html.items.len,
230 decl_table_html.items.ptr,
231 decl_table_html.items.len,
232 hdr.flags.use_llvm,
233 );
234}
lib/compiler/build_runner.zig+126-84
......@@ -9,7 +9,7 @@ const ArrayList = std.ArrayList;
99const File = std.fs.File;
1010const Step = std.Build.Step;
1111const Watch = std.Build.Watch;
12const Fuzz = std.Build.Fuzz;
12const WebServer = std.Build.WebServer;
1313const Allocator = std.mem.Allocator;
1414const fatal = std.process.fatal;
1515const Writer = std.io.Writer;
......@@ -25,15 +25,16 @@ pub const std_options: std.Options = .{
2525};
2626
2727pub fn main() !void {
28 // Here we use an ArenaAllocator backed by a page allocator because a build is a short-lived,
29 // one shot program. We don't need to waste time freeing memory and finding places to squish
30 // bytes into. So we free everything all at once at the very end.
31 var single_threaded_arena = std.heap.ArenaAllocator.init(std.heap.page_allocator);
28 // The build runner is often short-lived, but thanks to `--watch` and `--webui`, that's not
29 // always the case. So, we do need a true gpa for some things.
30 var debug_gpa_state: std.heap.DebugAllocator(.{}) = .init;
31 defer _ = debug_gpa_state.deinit();
32 const gpa = debug_gpa_state.allocator();
33
34 // ...but we'll back our arena by `std.heap.page_allocator` for efficiency.
35 var single_threaded_arena: std.heap.ArenaAllocator = .init(std.heap.page_allocator);
3236 defer single_threaded_arena.deinit();
33
34 var thread_safe_arena: std.heap.ThreadSafeAllocator = .{
35 .child_allocator = single_threaded_arena.allocator(),
36 };
37 var thread_safe_arena: std.heap.ThreadSafeAllocator = .{ .child_allocator = single_threaded_arena.allocator() };
3738 const arena = thread_safe_arena.allocator();
3839
3940 const args = try process.argsAlloc(arena);
......@@ -81,6 +82,7 @@ pub fn main() !void {
8182 .query = .{},
8283 .result = try std.zig.system.resolveTargetQuery(.{}),
8384 },
85 .time_report = false,
8486 };
8587
8688 graph.cache.addPrefix(.{ .path = null, .handle = std.fs.cwd() });
......@@ -113,7 +115,7 @@ pub fn main() !void {
113115 var watch = false;
114116 var fuzz = false;
115117 var debounce_interval_ms: u16 = 50;
116 var listen_port: u16 = 0;
118 var webui_listen: ?std.net.Address = null;
117119
118120 while (nextArg(args, &arg_idx)) |arg| {
119121 if (mem.startsWith(u8, arg, "-Z")) {
......@@ -220,13 +222,13 @@ pub fn main() !void {
220222 next_arg, @errorName(err),
221223 });
222224 };
223 } else if (mem.eql(u8, arg, "--port")) {
224 const next_arg = nextArg(args, &arg_idx) orelse
225 fatalWithHint("expected u16 after '{s}'", .{arg});
226 listen_port = std.fmt.parseUnsigned(u16, next_arg, 10) catch |err| {
227 fatal("unable to parse port '{s}' as unsigned 16-bit integer: {s}\n", .{
228 next_arg, @errorName(err),
229 });
225 } else if (mem.eql(u8, arg, "--webui")) {
226 webui_listen = std.net.Address.parseIp("::1", 0) catch unreachable;
227 } else if (mem.startsWith(u8, arg, "--webui=")) {
228 const addr_str = arg["--webui=".len..];
229 if (std.mem.eql(u8, addr_str, "-")) fatal("web interface cannot listen on stdio", .{});
230 webui_listen = std.net.Address.parseIpAndPort(addr_str) catch |err| {
231 fatal("invalid web UI address '{s}': {s}", .{ addr_str, @errorName(err) });
230232 };
231233 } else if (mem.eql(u8, arg, "--debug-log")) {
232234 const next_arg = nextArgOrFatal(args, &arg_idx);
......@@ -267,8 +269,16 @@ pub fn main() !void {
267269 prominent_compile_errors = true;
268270 } else if (mem.eql(u8, arg, "--watch")) {
269271 watch = true;
272 } else if (mem.eql(u8, arg, "--time-report")) {
273 graph.time_report = true;
274 if (webui_listen == null) {
275 webui_listen = std.net.Address.parseIp("::1", 0) catch unreachable;
276 }
270277 } else if (mem.eql(u8, arg, "--fuzz")) {
271278 fuzz = true;
279 if (webui_listen == null) {
280 webui_listen = std.net.Address.parseIp("::1", 0) catch unreachable;
281 }
272282 } else if (mem.eql(u8, arg, "-fincremental")) {
273283 graph.incremental = true;
274284 } else if (mem.eql(u8, arg, "-fno-incremental")) {
......@@ -331,6 +341,10 @@ pub fn main() !void {
331341 }
332342 }
333343
344 if (webui_listen != null and watch) fatal(
345 \\the build system does not yet support combining '--webui' and '--watch'; consider omitting '--watch' in favour of the web UI "Rebuild" button
346 , .{});
347
334348 const stderr: std.fs.File = .stderr();
335349 const ttyconf = get_tty_conf(color, stderr);
336350 switch (ttyconf) {
......@@ -394,14 +408,16 @@ pub fn main() !void {
394408 }
395409
396410 var run: Run = .{
411 .gpa = gpa,
412
397413 .max_rss = max_rss,
398414 .max_rss_is_default = false,
399415 .max_rss_mutex = .{},
400416 .skip_oom_steps = skip_oom_steps,
401417 .watch = watch,
402 .fuzz = fuzz,
403 .memory_blocked_steps = std.ArrayList(*Step).init(arena),
404 .step_stack = .{},
418 .web_server = undefined, // set after `prepare`
419 .memory_blocked_steps = .empty,
420 .step_stack = .empty,
405421 .prominent_compile_errors = prominent_compile_errors,
406422
407423 .claimed_rss = 0,
......@@ -410,74 +426,81 @@ pub fn main() !void {
410426 .stderr = stderr,
411427 .thread_pool = undefined,
412428 };
429 defer {
430 run.memory_blocked_steps.deinit(gpa);
431 run.step_stack.deinit(gpa);
432 }
413433
414434 if (run.max_rss == 0) {
415435 run.max_rss = process.totalSystemMemory() catch std.math.maxInt(u64);
416436 run.max_rss_is_default = true;
417437 }
418438
419 const gpa = arena;
420 prepare(gpa, arena, builder, targets.items, &run, graph.random_seed) catch |err| switch (err) {
439 prepare(arena, builder, targets.items, &run, graph.random_seed) catch |err| switch (err) {
421440 error.UncleanExit => process.exit(1),
422441 else => return err,
423442 };
424443
425 var w: Watch = if (watch and Watch.have_impl) try Watch.init() else undefined;
444 var w: Watch = w: {
445 if (!watch) break :w undefined;
446 if (!Watch.have_impl) fatal("--watch not yet implemented for {s}", .{@tagName(builtin.os.tag)});
447 break :w try .init();
448 };
426449
427450 try run.thread_pool.init(thread_pool_options);
428451 defer run.thread_pool.deinit();
429452
453 run.web_server = if (webui_listen) |listen_address| .init(.{
454 .gpa = gpa,
455 .thread_pool = &run.thread_pool,
456 .graph = &graph,
457 .all_steps = run.step_stack.keys(),
458 .ttyconf = run.ttyconf,
459 .root_prog_node = main_progress_node,
460 .watch = watch,
461 .listen_address = listen_address,
462 }) else null;
463
464 if (run.web_server) |*ws| {
465 ws.start() catch |err| fatal("failed to start web server: {s}", .{@errorName(err)});
466 }
467
430468 rebuild: while (true) {
469 if (run.web_server) |*ws| ws.startBuild();
470
431471 runStepNames(
432 gpa,
433472 builder,
434473 targets.items,
435474 main_progress_node,
436475 &run,
437476 ) catch |err| switch (err) {
438477 error.UncleanExit => {
439 assert(!run.watch);
478 assert(!run.watch and run.web_server == null);
440479 process.exit(1);
441480 },
442481 else => return err,
443482 };
444 if (fuzz) {
445 if (builtin.single_threaded) {
446 fatal("--fuzz not yet implemented for single-threaded builds", .{});
447 }
448 switch (builtin.os.tag) {
449 // Current implementation depends on two things that need to be ported to Windows:
450 // * Memory-mapping to share data between the fuzzer and build runner.
451 // * COFF/PE support added to `std.debug.Info` (it needs a batching API for resolving
452 // many addresses to source locations).
453 .windows => fatal("--fuzz not yet implemented for {s}", .{@tagName(builtin.os.tag)}),
454 else => {},
455 }
456 if (@bitSizeOf(usize) != 64) {
457 // Current implementation depends on posix.mmap()'s second parameter, `length: usize`,
458 // being compatible with `std.fs.getEndPos() u64`'s return value. This is not the case
459 // on 32-bit platforms.
460 // Affects or affected by issues #5185, #22523, and #22464.
461 fatal("--fuzz not yet implemented on {d}-bit platforms", .{@bitSizeOf(usize)});
462 }
463 const listen_address = std.net.Address.parseIp("127.0.0.1", listen_port) catch unreachable;
464 try Fuzz.start(
465 gpa,
466 arena,
467 global_cache_directory,
468 zig_lib_directory,
469 zig_exe,
470 &run.thread_pool,
471 run.step_stack.keys(),
472 run.ttyconf,
473 listen_address,
474 main_progress_node,
475 );
483
484 if (run.web_server) |*web_server| {
485 web_server.finishBuild(.{ .fuzz = fuzz });
476486 }
477487
478 if (!watch) return cleanExit();
488 if (!watch and run.web_server == null) {
489 return cleanExit();
490 }
479491
480 if (!Watch.have_impl) fatal("--watch not yet implemented for {s}", .{@tagName(builtin.os.tag)});
492 if (run.web_server) |*ws| {
493 assert(!watch); // fatal error after CLI parsing
494 while (true) switch (ws.wait()) {
495 .rebuild => {
496 for (run.step_stack.keys()) |step| {
497 step.state = .precheck_done;
498 step.reset(gpa);
499 }
500 continue :rebuild;
501 },
502 };
503 }
481504
482505 try w.update(gpa, run.step_stack.keys());
483506
......@@ -491,15 +514,16 @@ pub fn main() !void {
491514 w.dir_table.entries.len, countSubProcesses(run.step_stack.keys()),
492515 }) catch &caption_buf;
493516 var debouncing_node = main_progress_node.start(caption, 0);
494 var debounce_timeout: Watch.Timeout = .none;
495 while (true) switch (try w.wait(gpa, debounce_timeout)) {
517 var in_debounce = false;
518 while (true) switch (try w.wait(gpa, if (in_debounce) .{ .ms = debounce_interval_ms } else .none)) {
496519 .timeout => {
520 assert(in_debounce);
497521 debouncing_node.end();
498522 markFailedStepsDirty(gpa, run.step_stack.keys());
499523 continue :rebuild;
500524 },
501 .dirty => if (debounce_timeout == .none) {
502 debounce_timeout = .{ .ms = debounce_interval_ms };
525 .dirty => if (!in_debounce) {
526 in_debounce = true;
503527 debouncing_node.end();
504528 debouncing_node = main_progress_node.start("Debouncing (Change Detected)", 0);
505529 },
......@@ -530,13 +554,16 @@ fn countSubProcesses(all_steps: []const *Step) usize {
530554}
531555
532556const Run = struct {
557 gpa: Allocator,
533558 max_rss: u64,
534559 max_rss_is_default: bool,
535560 max_rss_mutex: std.Thread.Mutex,
536561 skip_oom_steps: bool,
537562 watch: bool,
538 fuzz: bool,
539 memory_blocked_steps: std.ArrayList(*Step),
563 web_server: ?WebServer,
564 /// Allocated into `gpa`.
565 memory_blocked_steps: std.ArrayListUnmanaged(*Step),
566 /// Allocated into `gpa`.
540567 step_stack: std.AutoArrayHashMapUnmanaged(*Step, void),
541568 prominent_compile_errors: bool,
542569 thread_pool: std.Thread.Pool,
......@@ -547,19 +574,19 @@ const Run = struct {
547574 stderr: File,
548575
549576 fn cleanExit(run: Run) void {
550 if (run.watch or run.fuzz) return;
577 if (run.watch or run.web_server != null) return;
551578 return runner.cleanExit();
552579 }
553580};
554581
555582fn prepare(
556 gpa: Allocator,
557583 arena: Allocator,
558584 b: *std.Build,
559585 step_names: []const []const u8,
560586 run: *Run,
561587 seed: u32,
562588) !void {
589 const gpa = run.gpa;
563590 const step_stack = &run.step_stack;
564591
565592 if (step_names.len == 0) {
......@@ -583,7 +610,7 @@ fn prepare(
583610 rand.shuffle(*Step, starting_steps);
584611
585612 for (starting_steps) |s| {
586 constructGraphAndCheckForDependencyLoop(b, s, &run.step_stack, rand) catch |err| switch (err) {
613 constructGraphAndCheckForDependencyLoop(gpa, b, s, &run.step_stack, rand) catch |err| switch (err) {
587614 error.DependencyLoopDetected => return uncleanExit(),
588615 else => |e| return e,
589616 };
......@@ -614,12 +641,12 @@ fn prepare(
614641}
615642
616643fn runStepNames(
617 gpa: Allocator,
618644 b: *std.Build,
619645 step_names: []const []const u8,
620646 parent_prog_node: std.Progress.Node,
621647 run: *Run,
622648) !void {
649 const gpa = run.gpa;
623650 const step_stack = &run.step_stack;
624651 const thread_pool = &run.thread_pool;
625652
......@@ -675,6 +702,7 @@ fn runStepNames(
675702 // B will be marked as dependency_failure, while A may never be queued, and thus
676703 // remain in the initial state of precheck_done.
677704 s.state = .dependency_failure;
705 if (run.web_server) |*ws| ws.updateStepStatus(s, .failure);
678706 pending_count += 1;
679707 },
680708 .dependency_failure => pending_count += 1,
......@@ -768,7 +796,7 @@ fn runStepNames(
768796 }
769797 }
770798
771 if (!run.watch) {
799 if (!run.watch and run.web_server == null) {
772800 // Signal to parent process that we have printed compile errors. The
773801 // parent process may choose to omit the "following command failed"
774802 // line in this case.
......@@ -777,7 +805,7 @@ fn runStepNames(
777805 }
778806 }
779807
780 if (!run.watch) return uncleanExit();
808 if (!run.watch and run.web_server == null) return uncleanExit();
781809}
782810
783811const PrintNode = struct {
......@@ -1022,6 +1050,7 @@ fn printTreeStep(
10221050/// when it finishes executing in `workerMakeOneStep`, it spawns next steps
10231051/// to run in random order
10241052fn constructGraphAndCheckForDependencyLoop(
1053 gpa: Allocator,
10251054 b: *std.Build,
10261055 s: *Step,
10271056 step_stack: *std.AutoArrayHashMapUnmanaged(*Step, void),
......@@ -1035,17 +1064,19 @@ fn constructGraphAndCheckForDependencyLoop(
10351064 .precheck_unstarted => {
10361065 s.state = .precheck_started;
10371066
1038 try step_stack.ensureUnusedCapacity(b.allocator, s.dependencies.items.len);
1067 try step_stack.ensureUnusedCapacity(gpa, s.dependencies.items.len);
10391068
10401069 // We dupe to avoid shuffling the steps in the summary, it depends
10411070 // on s.dependencies' order.
1042 const deps = b.allocator.dupe(*Step, s.dependencies.items) catch @panic("OOM");
1071 const deps = gpa.dupe(*Step, s.dependencies.items) catch @panic("OOM");
1072 defer gpa.free(deps);
1073
10431074 rand.shuffle(*Step, deps);
10441075
10451076 for (deps) |dep| {
1046 try step_stack.put(b.allocator, dep, {});
1077 try step_stack.put(gpa, dep, {});
10471078 try dep.dependants.append(b.allocator, s);
1048 constructGraphAndCheckForDependencyLoop(b, dep, step_stack, rand) catch |err| {
1079 constructGraphAndCheckForDependencyLoop(gpa, b, dep, step_stack, rand) catch |err| {
10491080 if (err == error.DependencyLoopDetected) {
10501081 std.debug.print(" {s}\n", .{s.name});
10511082 }
......@@ -1084,6 +1115,7 @@ fn workerMakeOneStep(
10841115 .success, .skipped => continue,
10851116 .failure, .dependency_failure, .skipped_oom => {
10861117 @atomicStore(Step.State, &s.state, .dependency_failure, .seq_cst);
1118 if (run.web_server) |*ws| ws.updateStepStatus(s, .failure);
10871119 return;
10881120 },
10891121 .precheck_done, .running => {
......@@ -1109,7 +1141,7 @@ fn workerMakeOneStep(
11091141 if (new_claimed_rss > run.max_rss) {
11101142 // Running this step right now could possibly exceed the allotted RSS.
11111143 // Add this step to the queue of memory-blocked steps.
1112 run.memory_blocked_steps.append(s) catch @panic("OOM");
1144 run.memory_blocked_steps.append(run.gpa, s) catch @panic("OOM");
11131145 return;
11141146 }
11151147
......@@ -1126,10 +1158,14 @@ fn workerMakeOneStep(
11261158 const sub_prog_node = prog_node.start(s.name, 0);
11271159 defer sub_prog_node.end();
11281160
1161 if (run.web_server) |*ws| ws.updateStepStatus(s, .wip);
1162
11291163 const make_result = s.make(.{
11301164 .progress_node = sub_prog_node,
11311165 .thread_pool = thread_pool,
11321166 .watch = run.watch,
1167 .web_server = if (run.web_server) |*ws| ws else null,
1168 .gpa = run.gpa,
11331169 });
11341170
11351171 // No matter the result, we want to display error/warning messages.
......@@ -1141,21 +1177,24 @@ fn workerMakeOneStep(
11411177 if (show_error_msgs or show_compile_errors or show_stderr) {
11421178 const bw = std.debug.lockStderrWriter(&stdio_buffer_allocation);
11431179 defer std.debug.unlockStderrWriter();
1144
1145 const gpa = b.allocator;
1146 printErrorMessages(gpa, s, .{ .ttyconf = run.ttyconf }, bw, run.prominent_compile_errors) catch {};
1180 printErrorMessages(run.gpa, s, .{ .ttyconf = run.ttyconf }, bw, run.prominent_compile_errors) catch {};
11471181 }
11481182
11491183 handle_result: {
11501184 if (make_result) |_| {
11511185 @atomicStore(Step.State, &s.state, .success, .seq_cst);
1186 if (run.web_server) |*ws| ws.updateStepStatus(s, .success);
11521187 } else |err| switch (err) {
11531188 error.MakeFailed => {
11541189 @atomicStore(Step.State, &s.state, .failure, .seq_cst);
1190 if (run.web_server) |*ws| ws.updateStepStatus(s, .failure);
11551191 std.Progress.setStatus(.failure_working);
11561192 break :handle_result;
11571193 },
1158 error.MakeSkipped => @atomicStore(Step.State, &s.state, .skipped, .seq_cst),
1194 error.MakeSkipped => {
1195 @atomicStore(Step.State, &s.state, .skipped, .seq_cst);
1196 if (run.web_server) |*ws| ws.updateStepStatus(s, .success);
1197 },
11591198 }
11601199
11611200 // Successful completion of a step, so we queue up its dependants as well.
......@@ -1255,10 +1294,10 @@ pub fn printErrorMessages(
12551294}
12561295
12571296fn printSteps(builder: *std.Build, w: *Writer) !void {
1258 const allocator = builder.allocator;
1297 const arena = builder.graph.arena;
12591298 for (builder.top_level_steps.values()) |top_level_step| {
12601299 const name = if (&top_level_step.step == builder.default_step)
1261 try fmt.allocPrint(allocator, "{s} (default)", .{top_level_step.step.name})
1300 try fmt.allocPrint(arena, "{s} (default)", .{top_level_step.step.name})
12621301 else
12631302 top_level_step.step.name;
12641303 try w.print(" {s:<28} {s}\n", .{ name, top_level_step.description });
......@@ -1319,8 +1358,11 @@ fn printUsage(b: *std.Build, w: *Writer) !void {
13191358 \\ needed (Default) Lazy dependencies are fetched as needed
13201359 \\ all Lazy dependencies are always fetched
13211360 \\ --watch Continuously rebuild when source files are modified
1322 \\ --fuzz Continuously search for unit test failures
13231361 \\ --debounce <ms> Delay before rebuilding after changed file detected
1362 \\ --webui[=ip] Enable the web interface on the given IP address
1363 \\ --fuzz Continuously search for unit test failures (implies '--webui')
1364 \\ --time-report Force full rebuild and provide detailed information on
1365 \\ compilation time of Zig source code (implies '--webui')
13241366 \\ -fincremental Enable incremental compilation
13251367 \\ -fno-incremental Disable incremental compilation
13261368 \\
......@@ -1328,7 +1370,7 @@ fn printUsage(b: *std.Build, w: *Writer) !void {
13281370 \\
13291371 );
13301372
1331 const arena = b.allocator;
1373 const arena = b.graph.arena;
13321374 if (b.available_options_list.items.len == 0) {
13331375 try w.print(" (none)\n", .{});
13341376 } else {
lib/fuzzer.zig+1-1
......@@ -3,7 +3,7 @@ const std = @import("std");
33const Allocator = std.mem.Allocator;
44const assert = std.debug.assert;
55const fatal = std.process.fatal;
6const SeenPcsHeader = std.Build.Fuzz.abi.SeenPcsHeader;
6const SeenPcsHeader = std.Build.abi.fuzz.SeenPcsHeader;
77
88pub const std_options = std.Options{
99 .logFn = logOverride,
lib/fuzzer/web/index.html deleted-161
......@@ -1,161 +0,0 @@
1<!doctype html>
2<html>
3 <head>
4 <meta charset="utf-8">
5 <title>Zig Build System Interface</title>
6 <style type="text/css">
7 body {
8 font-family: system-ui, -apple-system, Roboto, "Segoe UI", sans-serif;
9 color: #000000;
10 }
11 .hidden {
12 display: none;
13 }
14 table {
15 width: 100%;
16 }
17 a {
18 color: #2A6286;
19 }
20 pre{
21 font-family:"Source Code Pro",monospace;
22 font-size:1em;
23 background-color:#F5F5F5;
24 padding: 1em;
25 margin: 0;
26 overflow-x: auto;
27 }
28 :not(pre) > code {
29 white-space: break-spaces;
30 }
31 code {
32 font-family:"Source Code Pro",monospace;
33 font-size: 0.9em;
34 }
35 code a {
36 color: #000000;
37 }
38 kbd {
39 color: #000;
40 background-color: #fafbfc;
41 border-color: #d1d5da;
42 border-bottom-color: #c6cbd1;
43 box-shadow-color: #c6cbd1;
44 display: inline-block;
45 padding: 0.3em 0.2em;
46 font: 1.2em monospace;
47 line-height: 0.8em;
48 vertical-align: middle;
49 border: solid 1px;
50 border-radius: 3px;
51 box-shadow: inset 0 -1px 0;
52 cursor: default;
53 }
54
55 .l {
56 display: inline-block;
57 background: red;
58 width: 1em;
59 height: 1em;
60 border-radius: 1em;
61 }
62 .c {
63 background-color: green;
64 }
65
66 .tok-kw {
67 color: #333;
68 font-weight: bold;
69 }
70 .tok-str {
71 color: #d14;
72 }
73 .tok-builtin {
74 color: #0086b3;
75 }
76 .tok-comment {
77 color: #777;
78 font-style: italic;
79 }
80 .tok-fn {
81 color: #900;
82 font-weight: bold;
83 }
84 .tok-null {
85 color: #008080;
86 }
87 .tok-number {
88 color: #008080;
89 }
90 .tok-type {
91 color: #458;
92 font-weight: bold;
93 }
94
95 @media (prefers-color-scheme: dark) {
96 body {
97 background-color: #111;
98 color: #bbb;
99 }
100 pre {
101 background-color: #222;
102 color: #ccc;
103 }
104 a {
105 color: #88f;
106 }
107 code a {
108 color: #ccc;
109 }
110 .l {
111 background-color: red;
112 }
113 .c {
114 background-color: green;
115 }
116 .tok-kw {
117 color: #eee;
118 }
119 .tok-str {
120 color: #2e5;
121 }
122 .tok-builtin {
123 color: #ff894c;
124 }
125 .tok-comment {
126 color: #aa7;
127 }
128 .tok-fn {
129 color: #B1A0F8;
130 }
131 .tok-null {
132 color: #ff8080;
133 }
134 .tok-number {
135 color: #ff8080;
136 }
137 .tok-type {
138 color: #68f;
139 }
140 }
141 </style>
142 </head>
143 <body>
144 <p id="status">Loading JavaScript...</p>
145 <div id="sectStats" class="hidden">
146 <ul>
147 <li>Total Runs: <span id="statTotalRuns"></span></li>
148 <li>Unique Runs: <span id="statUniqueRuns"></span></li>
149 <li>Speed (Runs/Second): <span id="statSpeed"></span></li>
150 <li>Coverage: <span id="statCoverage"></span></li>
151 <li>Entry Points: <ul id="entryPointsList"></ul></li>
152 </ul>
153 </div>
154 <div id="sectSource" class="hidden">
155 <h2>Source Code</h2>
156 <pre><code id="sourceText"></code></pre>
157 </div>
158 <script src="main.js"></script>
159 </body>
160</html>
161
lib/fuzzer/web/main.js deleted-252
......@@ -1,252 +0,0 @@
1(function() {
2 const domStatus = document.getElementById("status");
3 const domSectSource = document.getElementById("sectSource");
4 const domSectStats = document.getElementById("sectStats");
5 const domSourceText = document.getElementById("sourceText");
6 const domStatTotalRuns = document.getElementById("statTotalRuns");
7 const domStatUniqueRuns = document.getElementById("statUniqueRuns");
8 const domStatSpeed = document.getElementById("statSpeed");
9 const domStatCoverage = document.getElementById("statCoverage");
10 const domEntryPointsList = document.getElementById("entryPointsList");
11
12 let wasm_promise = fetch("main.wasm");
13 let sources_promise = fetch("sources.tar").then(function(response) {
14 if (!response.ok) throw new Error("unable to download sources");
15 return response.arrayBuffer();
16 });
17 var wasm_exports = null;
18 var curNavSearch = null;
19 var curNavLocation = null;
20
21 const text_decoder = new TextDecoder();
22 const text_encoder = new TextEncoder();
23
24 domStatus.textContent = "Loading WebAssembly...";
25 WebAssembly.instantiateStreaming(wasm_promise, {
26 js: {
27 log: function(ptr, len) {
28 const msg = decodeString(ptr, len);
29 console.log(msg);
30 },
31 panic: function (ptr, len) {
32 const msg = decodeString(ptr, len);
33 throw new Error("panic: " + msg);
34 },
35 timestamp: function () {
36 return BigInt(new Date());
37 },
38 emitSourceIndexChange: onSourceIndexChange,
39 emitCoverageUpdate: onCoverageUpdate,
40 emitEntryPointsUpdate: renderStats,
41 },
42 }).then(function(obj) {
43 wasm_exports = obj.instance.exports;
44 window.wasm = obj; // for debugging
45 domStatus.textContent = "Loading sources tarball...";
46
47 sources_promise.then(function(buffer) {
48 domStatus.textContent = "Parsing sources...";
49 const js_array = new Uint8Array(buffer);
50 const ptr = wasm_exports.alloc(js_array.length);
51 const wasm_array = new Uint8Array(wasm_exports.memory.buffer, ptr, js_array.length);
52 wasm_array.set(js_array);
53 wasm_exports.unpack(ptr, js_array.length);
54
55 window.addEventListener('popstate', onPopState, false);
56 onHashChange(null);
57
58 domStatus.textContent = "Waiting for server to send source location metadata...";
59 connectWebSocket();
60 });
61 });
62
63 function onPopState(ev) {
64 onHashChange(ev.state);
65 }
66
67 function onHashChange(state) {
68 history.replaceState({}, "");
69 navigate(location.hash);
70 if (state == null) window.scrollTo({top: 0});
71 }
72
73 function navigate(location_hash) {
74 domSectSource.classList.add("hidden");
75
76 curNavLocation = null;
77 curNavSearch = null;
78
79 if (location_hash.length > 1 && location_hash[0] === '#') {
80 const query = location_hash.substring(1);
81 const qpos = query.indexOf("?");
82 let nonSearchPart;
83 if (qpos === -1) {
84 nonSearchPart = query;
85 } else {
86 nonSearchPart = query.substring(0, qpos);
87 curNavSearch = decodeURIComponent(query.substring(qpos + 1));
88 }
89
90 if (nonSearchPart[0] == "l") {
91 curNavLocation = +nonSearchPart.substring(1);
92 renderSource(curNavLocation);
93 }
94 }
95
96 render();
97 }
98
99 function connectWebSocket() {
100 const host = document.location.host;
101 const pathname = document.location.pathname;
102 const isHttps = document.location.protocol === 'https:';
103 const match = host.match(/^(.+):(\d+)$/);
104 const defaultPort = isHttps ? 443 : 80;
105 const port = match ? parseInt(match[2], 10) : defaultPort;
106 const hostName = match ? match[1] : host;
107 const wsProto = isHttps ? "wss:" : "ws:";
108 const wsUrl = wsProto + '//' + hostName + ':' + port + pathname;
109 ws = new WebSocket(wsUrl);
110 ws.binaryType = "arraybuffer";
111 ws.addEventListener('message', onWebSocketMessage, false);
112 ws.addEventListener('error', timeoutThenCreateNew, false);
113 ws.addEventListener('close', timeoutThenCreateNew, false);
114 ws.addEventListener('open', onWebSocketOpen, false);
115 }
116
117 function onWebSocketOpen() {
118 //console.log("web socket opened");
119 }
120
121 function onWebSocketMessage(ev) {
122 wasmOnMessage(ev.data);
123 }
124
125 function timeoutThenCreateNew() {
126 ws.removeEventListener('message', onWebSocketMessage, false);
127 ws.removeEventListener('error', timeoutThenCreateNew, false);
128 ws.removeEventListener('close', timeoutThenCreateNew, false);
129 ws.removeEventListener('open', onWebSocketOpen, false);
130 ws = null;
131 setTimeout(connectWebSocket, 1000);
132 }
133
134 function wasmOnMessage(data) {
135 const jsArray = new Uint8Array(data);
136 const ptr = wasm_exports.message_begin(jsArray.length);
137 const wasmArray = new Uint8Array(wasm_exports.memory.buffer, ptr, jsArray.length);
138 wasmArray.set(jsArray);
139 wasm_exports.message_end();
140 }
141
142 function onSourceIndexChange() {
143 render();
144 if (curNavLocation != null) renderSource(curNavLocation);
145 }
146
147 function onCoverageUpdate() {
148 renderStats();
149 renderCoverage();
150 }
151
152 function render() {
153 domStatus.classList.add("hidden");
154 }
155
156 function renderStats() {
157 const totalRuns = wasm_exports.totalRuns();
158 const uniqueRuns = wasm_exports.uniqueRuns();
159 const totalSourceLocations = wasm_exports.totalSourceLocations();
160 const coveredSourceLocations = wasm_exports.coveredSourceLocations();
161 domStatTotalRuns.innerText = totalRuns;
162 domStatUniqueRuns.innerText = uniqueRuns + " (" + percent(uniqueRuns, totalRuns) + "%)";
163 domStatCoverage.innerText = coveredSourceLocations + " / " + totalSourceLocations + " (" + percent(coveredSourceLocations, totalSourceLocations) + "%)";
164 domStatSpeed.innerText = wasm_exports.totalRunsPerSecond().toFixed(0);
165
166 const entryPoints = unwrapInt32Array(wasm_exports.entryPoints());
167 resizeDomList(domEntryPointsList, entryPoints.length, "<li></li>");
168 for (let i = 0; i < entryPoints.length; i += 1) {
169 const liDom = domEntryPointsList.children[i];
170 liDom.innerHTML = unwrapString(wasm_exports.sourceLocationLinkHtml(entryPoints[i]));
171 }
172
173
174 domSectStats.classList.remove("hidden");
175 }
176
177 function renderCoverage() {
178 if (curNavLocation == null) return;
179 const sourceLocationIndex = curNavLocation;
180
181 for (let i = 0; i < domSourceText.children.length; i += 1) {
182 const childDom = domSourceText.children[i];
183 if (childDom.id != null && childDom.id[0] == "l") {
184 childDom.classList.add("l");
185 childDom.classList.remove("c");
186 }
187 }
188 const coveredList = unwrapInt32Array(wasm_exports.sourceLocationFileCoveredList(sourceLocationIndex));
189 for (let i = 0; i < coveredList.length; i += 1) {
190 document.getElementById("l" + coveredList[i]).classList.add("c");
191 }
192 }
193
194 function resizeDomList(listDom, desiredLen, templateHtml) {
195 for (let i = listDom.childElementCount; i < desiredLen; i += 1) {
196 listDom.insertAdjacentHTML('beforeend', templateHtml);
197 }
198 while (desiredLen < listDom.childElementCount) {
199 listDom.removeChild(listDom.lastChild);
200 }
201 }
202
203 function percent(a, b) {
204 return ((Number(a) / Number(b)) * 100).toFixed(1);
205 }
206
207 function renderSource(sourceLocationIndex) {
208 const pathName = unwrapString(wasm_exports.sourceLocationPath(sourceLocationIndex));
209 if (pathName.length === 0) return;
210
211 const h2 = domSectSource.children[0];
212 h2.innerText = pathName;
213 domSourceText.innerHTML = unwrapString(wasm_exports.sourceLocationFileHtml(sourceLocationIndex));
214
215 domSectSource.classList.remove("hidden");
216
217 // Empirically, Firefox needs this requestAnimationFrame in order for the scrollIntoView to work.
218 requestAnimationFrame(function() {
219 const slDom = document.getElementById("l" + sourceLocationIndex);
220 if (slDom != null) slDom.scrollIntoView({
221 behavior: "smooth",
222 block: "center",
223 });
224 });
225 }
226
227 function decodeString(ptr, len) {
228 if (len === 0) return "";
229 return text_decoder.decode(new Uint8Array(wasm_exports.memory.buffer, ptr, len));
230 }
231
232 function unwrapInt32Array(bigint) {
233 const ptr = Number(bigint & 0xffffffffn);
234 const len = Number(bigint >> 32n);
235 if (len === 0) return new Uint32Array();
236 return new Uint32Array(wasm_exports.memory.buffer, ptr, len);
237 }
238
239 function setInputString(s) {
240 const jsArray = text_encoder.encode(s);
241 const len = jsArray.length;
242 const ptr = wasm_exports.set_input_string(len);
243 const wasmArray = new Uint8Array(wasm_exports.memory.buffer, ptr, len);
244 wasmArray.set(jsArray);
245 }
246
247 function unwrapString(bigint) {
248 const ptr = Number(bigint & 0xffffffffn);
249 const len = Number(bigint >> 32n);
250 return decodeString(ptr, len);
251 }
252})();
lib/fuzzer/web/main.zig deleted-455
......@@ -1,455 +0,0 @@
1const std = @import("std");
2const assert = std.debug.assert;
3const abi = std.Build.Fuzz.abi;
4const gpa = std.heap.wasm_allocator;
5const log = std.log;
6const Coverage = std.debug.Coverage;
7const Allocator = std.mem.Allocator;
8
9const Walk = @import("Walk");
10const Decl = Walk.Decl;
11const html_render = @import("html_render");
12
13/// Nanoseconds.
14var server_base_timestamp: i64 = 0;
15/// Milliseconds.
16var client_base_timestamp: i64 = 0;
17/// Relative to `server_base_timestamp`.
18var start_fuzzing_timestamp: i64 = undefined;
19
20const js = struct {
21 extern "js" fn log(ptr: [*]const u8, len: usize) void;
22 extern "js" fn panic(ptr: [*]const u8, len: usize) noreturn;
23 extern "js" fn timestamp() i64;
24 extern "js" fn emitSourceIndexChange() void;
25 extern "js" fn emitCoverageUpdate() void;
26 extern "js" fn emitEntryPointsUpdate() void;
27};
28
29pub const std_options: std.Options = .{
30 .logFn = logFn,
31};
32
33pub fn panic(msg: []const u8, st: ?*std.builtin.StackTrace, addr: ?usize) noreturn {
34 _ = st;
35 _ = addr;
36 log.err("panic: {s}", .{msg});
37 @trap();
38}
39
40fn logFn(
41 comptime message_level: log.Level,
42 comptime scope: @TypeOf(.enum_literal),
43 comptime format: []const u8,
44 args: anytype,
45) void {
46 const level_txt = comptime message_level.asText();
47 const prefix2 = if (scope == .default) ": " else "(" ++ @tagName(scope) ++ "): ";
48 var buf: [500]u8 = undefined;
49 const line = std.fmt.bufPrint(&buf, level_txt ++ prefix2 ++ format, args) catch l: {
50 buf[buf.len - 3 ..][0..3].* = "...".*;
51 break :l &buf;
52 };
53 js.log(line.ptr, line.len);
54}
55
56export fn alloc(n: usize) [*]u8 {
57 const slice = gpa.alloc(u8, n) catch @panic("OOM");
58 return slice.ptr;
59}
60
61var message_buffer: std.ArrayListAlignedUnmanaged(u8, .of(u64)) = .empty;
62
63/// Resizes the message buffer to be the correct length; returns the pointer to
64/// the query string.
65export fn message_begin(len: usize) [*]u8 {
66 message_buffer.resize(gpa, len) catch @panic("OOM");
67 return message_buffer.items.ptr;
68}
69
70export fn message_end() void {
71 const msg_bytes = message_buffer.items;
72
73 const tag: abi.ToClientTag = @enumFromInt(msg_bytes[0]);
74 switch (tag) {
75 .current_time => return currentTimeMessage(msg_bytes),
76 .source_index => return sourceIndexMessage(msg_bytes) catch @panic("OOM"),
77 .coverage_update => return coverageUpdateMessage(msg_bytes) catch @panic("OOM"),
78 .entry_points => return entryPointsMessage(msg_bytes) catch @panic("OOM"),
79 _ => unreachable,
80 }
81}
82
83export fn unpack(tar_ptr: [*]u8, tar_len: usize) void {
84 const tar_bytes = tar_ptr[0..tar_len];
85 log.debug("received {d} bytes of tar file", .{tar_bytes.len});
86
87 unpackInner(tar_bytes) catch |err| {
88 fatal("unable to unpack tar: {s}", .{@errorName(err)});
89 };
90}
91
92/// Set by `set_input_string`.
93var input_string: std.ArrayListUnmanaged(u8) = .empty;
94var string_result: std.ArrayListUnmanaged(u8) = .empty;
95
96export fn set_input_string(len: usize) [*]u8 {
97 input_string.resize(gpa, len) catch @panic("OOM");
98 return input_string.items.ptr;
99}
100
101/// Looks up the root struct decl corresponding to a file by path.
102/// Uses `input_string`.
103export fn find_file_root() Decl.Index {
104 const file: Walk.File.Index = @enumFromInt(Walk.files.getIndex(input_string.items) orelse return .none);
105 return file.findRootDecl();
106}
107
108export fn decl_source_html(decl_index: Decl.Index) String {
109 const decl = decl_index.get();
110
111 string_result.clearRetainingCapacity();
112 html_render.fileSourceHtml(decl.file, &string_result, decl.ast_node, .{}) catch |err| {
113 fatal("unable to render source: {s}", .{@errorName(err)});
114 };
115 return String.init(string_result.items);
116}
117
118export fn totalSourceLocations() usize {
119 return coverage_source_locations.items.len;
120}
121
122export fn coveredSourceLocations() usize {
123 const covered_bits = recent_coverage_update.items[@sizeOf(abi.CoverageUpdateHeader)..];
124 var count: usize = 0;
125 for (covered_bits) |byte| count += @popCount(byte);
126 return count;
127}
128
129fn getCoverageUpdateHeader() *abi.CoverageUpdateHeader {
130 return @alignCast(@ptrCast(recent_coverage_update.items[0..@sizeOf(abi.CoverageUpdateHeader)]));
131}
132
133export fn totalRuns() u64 {
134 const header = getCoverageUpdateHeader();
135 return header.n_runs;
136}
137
138export fn uniqueRuns() u64 {
139 const header = getCoverageUpdateHeader();
140 return header.unique_runs;
141}
142
143export fn totalRunsPerSecond() f64 {
144 @setFloatMode(.optimized);
145 const header = getCoverageUpdateHeader();
146 const ns_elapsed: f64 = @floatFromInt(nsSince(start_fuzzing_timestamp));
147 const n_runs: f64 = @floatFromInt(header.n_runs);
148 return n_runs / (ns_elapsed / std.time.ns_per_s);
149}
150
151const String = Slice(u8);
152
153fn Slice(T: type) type {
154 return packed struct(u64) {
155 ptr: u32,
156 len: u32,
157
158 fn init(s: []const T) @This() {
159 return .{
160 .ptr = @intFromPtr(s.ptr),
161 .len = s.len,
162 };
163 }
164 };
165}
166
167fn unpackInner(tar_bytes: []u8) !void {
168 var fbs = std.io.fixedBufferStream(tar_bytes);
169 var file_name_buffer: [1024]u8 = undefined;
170 var link_name_buffer: [1024]u8 = undefined;
171 var it = std.tar.iterator(fbs.reader(), .{
172 .file_name_buffer = &file_name_buffer,
173 .link_name_buffer = &link_name_buffer,
174 });
175 while (try it.next()) |tar_file| {
176 switch (tar_file.kind) {
177 .file => {
178 if (tar_file.size == 0 and tar_file.name.len == 0) break;
179 if (std.mem.endsWith(u8, tar_file.name, ".zig")) {
180 log.debug("found file: '{s}'", .{tar_file.name});
181 const file_name = try gpa.dupe(u8, tar_file.name);
182 if (std.mem.indexOfScalar(u8, file_name, '/')) |pkg_name_end| {
183 const pkg_name = file_name[0..pkg_name_end];
184 const gop = try Walk.modules.getOrPut(gpa, pkg_name);
185 const file: Walk.File.Index = @enumFromInt(Walk.files.entries.len);
186 if (!gop.found_existing or
187 std.mem.eql(u8, file_name[pkg_name_end..], "/root.zig") or
188 std.mem.eql(u8, file_name[pkg_name_end + 1 .. file_name.len - ".zig".len], pkg_name))
189 {
190 gop.value_ptr.* = file;
191 }
192 const file_bytes = tar_bytes[fbs.pos..][0..@intCast(tar_file.size)];
193 assert(file == try Walk.add_file(file_name, file_bytes));
194 }
195 } else {
196 log.warn("skipping: '{s}' - the tar creation should have done that", .{tar_file.name});
197 }
198 },
199 else => continue,
200 }
201 }
202}
203
204fn fatal(comptime format: []const u8, args: anytype) noreturn {
205 var buf: [500]u8 = undefined;
206 const line = std.fmt.bufPrint(&buf, format, args) catch l: {
207 buf[buf.len - 3 ..][0..3].* = "...".*;
208 break :l &buf;
209 };
210 js.panic(line.ptr, line.len);
211}
212
213fn currentTimeMessage(msg_bytes: []u8) void {
214 client_base_timestamp = js.timestamp();
215 server_base_timestamp = @bitCast(msg_bytes[1..][0..8].*);
216}
217
218/// Nanoseconds passed since a server timestamp.
219fn nsSince(server_timestamp: i64) i64 {
220 const ms_passed = js.timestamp() - client_base_timestamp;
221 const ns_passed = server_base_timestamp - server_timestamp;
222 return ns_passed + ms_passed * std.time.ns_per_ms;
223}
224
225fn sourceIndexMessage(msg_bytes: []u8) error{OutOfMemory}!void {
226 const Header = abi.SourceIndexHeader;
227 const header: Header = @bitCast(msg_bytes[0..@sizeOf(Header)].*);
228
229 const directories_start = @sizeOf(Header);
230 const directories_end = directories_start + header.directories_len * @sizeOf(Coverage.String);
231 const files_start = directories_end;
232 const files_end = files_start + header.files_len * @sizeOf(Coverage.File);
233 const source_locations_start = files_end;
234 const source_locations_end = source_locations_start + header.source_locations_len * @sizeOf(Coverage.SourceLocation);
235 const string_bytes = msg_bytes[source_locations_end..][0..header.string_bytes_len];
236
237 const directories: []const Coverage.String = @alignCast(std.mem.bytesAsSlice(Coverage.String, msg_bytes[directories_start..directories_end]));
238 const files: []const Coverage.File = @alignCast(std.mem.bytesAsSlice(Coverage.File, msg_bytes[files_start..files_end]));
239 const source_locations: []const Coverage.SourceLocation = @alignCast(std.mem.bytesAsSlice(Coverage.SourceLocation, msg_bytes[source_locations_start..source_locations_end]));
240
241 start_fuzzing_timestamp = header.start_timestamp;
242 try updateCoverage(directories, files, source_locations, string_bytes);
243 js.emitSourceIndexChange();
244}
245
246fn coverageUpdateMessage(msg_bytes: []u8) error{OutOfMemory}!void {
247 recent_coverage_update.clearRetainingCapacity();
248 recent_coverage_update.appendSlice(gpa, msg_bytes) catch @panic("OOM");
249 js.emitCoverageUpdate();
250}
251
252var entry_points: std.ArrayListUnmanaged(u32) = .empty;
253
254fn entryPointsMessage(msg_bytes: []u8) error{OutOfMemory}!void {
255 const header: abi.EntryPointHeader = @bitCast(msg_bytes[0..@sizeOf(abi.EntryPointHeader)].*);
256 entry_points.resize(gpa, header.flags.locs_len) catch @panic("OOM");
257 @memcpy(entry_points.items, std.mem.bytesAsSlice(u32, msg_bytes[@sizeOf(abi.EntryPointHeader)..]));
258 js.emitEntryPointsUpdate();
259}
260
261export fn entryPoints() Slice(u32) {
262 return Slice(u32).init(entry_points.items);
263}
264
265/// Index into `coverage_source_locations`.
266const SourceLocationIndex = enum(u32) {
267 _,
268
269 fn haveCoverage(sli: SourceLocationIndex) bool {
270 return @intFromEnum(sli) < coverage_source_locations.items.len;
271 }
272
273 fn ptr(sli: SourceLocationIndex) *Coverage.SourceLocation {
274 return &coverage_source_locations.items[@intFromEnum(sli)];
275 }
276
277 fn sourceLocationLinkHtml(
278 sli: SourceLocationIndex,
279 out: *std.ArrayListUnmanaged(u8),
280 ) Allocator.Error!void {
281 const sl = sli.ptr();
282 try out.writer(gpa).print("<a href=\"#l{d}\">", .{@intFromEnum(sli)});
283 try sli.appendPath(out);
284 try out.writer(gpa).print(":{d}:{d}</a>", .{ sl.line, sl.column });
285 }
286
287 fn appendPath(sli: SourceLocationIndex, out: *std.ArrayListUnmanaged(u8)) Allocator.Error!void {
288 const sl = sli.ptr();
289 const file = coverage.fileAt(sl.file);
290 const file_name = coverage.stringAt(file.basename);
291 const dir_name = coverage.stringAt(coverage.directories.keys()[file.directory_index]);
292 try html_render.appendEscaped(out, dir_name);
293 try out.appendSlice(gpa, "/");
294 try html_render.appendEscaped(out, file_name);
295 }
296
297 fn toWalkFile(sli: SourceLocationIndex) ?Walk.File.Index {
298 var buf: std.ArrayListUnmanaged(u8) = .empty;
299 defer buf.deinit(gpa);
300 sli.appendPath(&buf) catch @panic("OOM");
301 return @enumFromInt(Walk.files.getIndex(buf.items) orelse return null);
302 }
303
304 fn fileHtml(
305 sli: SourceLocationIndex,
306 out: *std.ArrayListUnmanaged(u8),
307 ) error{ OutOfMemory, SourceUnavailable }!void {
308 const walk_file_index = sli.toWalkFile() orelse return error.SourceUnavailable;
309 const root_node = walk_file_index.findRootDecl().get().ast_node;
310 var annotations: std.ArrayListUnmanaged(html_render.Annotation) = .empty;
311 defer annotations.deinit(gpa);
312 try computeSourceAnnotations(sli.ptr().file, walk_file_index, &annotations, coverage_source_locations.items);
313 html_render.fileSourceHtml(walk_file_index, out, root_node, .{
314 .source_location_annotations = annotations.items,
315 }) catch |err| {
316 fatal("unable to render source: {s}", .{@errorName(err)});
317 };
318 }
319};
320
321fn computeSourceAnnotations(
322 cov_file_index: Coverage.File.Index,
323 walk_file_index: Walk.File.Index,
324 annotations: *std.ArrayListUnmanaged(html_render.Annotation),
325 source_locations: []const Coverage.SourceLocation,
326) !void {
327 // Collect all the source locations from only this file into this array
328 // first, then sort by line, col, so that we can collect annotations with
329 // O(N) time complexity.
330 var locs: std.ArrayListUnmanaged(SourceLocationIndex) = .empty;
331 defer locs.deinit(gpa);
332
333 for (source_locations, 0..) |sl, sli_usize| {
334 if (sl.file != cov_file_index) continue;
335 const sli: SourceLocationIndex = @enumFromInt(sli_usize);
336 try locs.append(gpa, sli);
337 }
338
339 std.mem.sortUnstable(SourceLocationIndex, locs.items, {}, struct {
340 pub fn lessThan(context: void, lhs: SourceLocationIndex, rhs: SourceLocationIndex) bool {
341 _ = context;
342 const lhs_ptr = lhs.ptr();
343 const rhs_ptr = rhs.ptr();
344 if (lhs_ptr.line < rhs_ptr.line) return true;
345 if (lhs_ptr.line > rhs_ptr.line) return false;
346 return lhs_ptr.column < rhs_ptr.column;
347 }
348 }.lessThan);
349
350 const source = walk_file_index.get_ast().source;
351 var line: usize = 1;
352 var column: usize = 1;
353 var next_loc_index: usize = 0;
354 for (source, 0..) |byte, offset| {
355 if (byte == '\n') {
356 line += 1;
357 column = 1;
358 } else {
359 column += 1;
360 }
361 while (true) {
362 if (next_loc_index >= locs.items.len) return;
363 const next_sli = locs.items[next_loc_index];
364 const next_sl = next_sli.ptr();
365 if (next_sl.line > line or (next_sl.line == line and next_sl.column >= column)) break;
366 try annotations.append(gpa, .{
367 .file_byte_offset = offset,
368 .dom_id = @intFromEnum(next_sli),
369 });
370 next_loc_index += 1;
371 }
372 }
373}
374
375var coverage = Coverage.init;
376/// Index of type `SourceLocationIndex`.
377var coverage_source_locations: std.ArrayListUnmanaged(Coverage.SourceLocation) = .empty;
378/// Contains the most recent coverage update message, unmodified.
379var recent_coverage_update: std.ArrayListAlignedUnmanaged(u8, .of(u64)) = .empty;
380
381fn updateCoverage(
382 directories: []const Coverage.String,
383 files: []const Coverage.File,
384 source_locations: []const Coverage.SourceLocation,
385 string_bytes: []const u8,
386) !void {
387 coverage.directories.clearRetainingCapacity();
388 coverage.files.clearRetainingCapacity();
389 coverage.string_bytes.clearRetainingCapacity();
390 coverage_source_locations.clearRetainingCapacity();
391
392 try coverage_source_locations.appendSlice(gpa, source_locations);
393 try coverage.string_bytes.appendSlice(gpa, string_bytes);
394
395 try coverage.files.entries.resize(gpa, files.len);
396 @memcpy(coverage.files.entries.items(.key), files);
397 try coverage.files.reIndexContext(gpa, .{ .string_bytes = coverage.string_bytes.items });
398
399 try coverage.directories.entries.resize(gpa, directories.len);
400 @memcpy(coverage.directories.entries.items(.key), directories);
401 try coverage.directories.reIndexContext(gpa, .{ .string_bytes = coverage.string_bytes.items });
402}
403
404export fn sourceLocationLinkHtml(index: SourceLocationIndex) String {
405 string_result.clearRetainingCapacity();
406 index.sourceLocationLinkHtml(&string_result) catch @panic("OOM");
407 return String.init(string_result.items);
408}
409
410/// Returns empty string if coverage metadata is not available for this source location.
411export fn sourceLocationPath(sli: SourceLocationIndex) String {
412 string_result.clearRetainingCapacity();
413 if (sli.haveCoverage()) sli.appendPath(&string_result) catch @panic("OOM");
414 return String.init(string_result.items);
415}
416
417export fn sourceLocationFileHtml(sli: SourceLocationIndex) String {
418 string_result.clearRetainingCapacity();
419 sli.fileHtml(&string_result) catch |err| switch (err) {
420 error.OutOfMemory => @panic("OOM"),
421 error.SourceUnavailable => {},
422 };
423 return String.init(string_result.items);
424}
425
426export fn sourceLocationFileCoveredList(sli_file: SourceLocationIndex) Slice(SourceLocationIndex) {
427 const global = struct {
428 var result: std.ArrayListUnmanaged(SourceLocationIndex) = .empty;
429 fn add(i: u32, want_file: Coverage.File.Index) void {
430 const src_loc_index: SourceLocationIndex = @enumFromInt(i);
431 if (src_loc_index.ptr().file == want_file) result.appendAssumeCapacity(src_loc_index);
432 }
433 };
434 const want_file = sli_file.ptr().file;
435 global.result.clearRetainingCapacity();
436
437 // This code assumes 64-bit elements, which is incorrect if the executable
438 // being fuzzed is not a 64-bit CPU. It also assumes little-endian which
439 // can also be incorrect.
440 comptime assert(abi.CoverageUpdateHeader.trailing[0] == .pc_bits_usize);
441 const n_bitset_elems = (coverage_source_locations.items.len + @bitSizeOf(u64) - 1) / @bitSizeOf(u64);
442 const covered_bits = std.mem.bytesAsSlice(
443 u64,
444 recent_coverage_update.items[@sizeOf(abi.CoverageUpdateHeader)..][0 .. n_bitset_elems * @sizeOf(u64)],
445 );
446 var sli: u32 = 0;
447 for (covered_bits) |elem| {
448 global.result.ensureUnusedCapacity(gpa, 64) catch @panic("OOM");
449 for (0..@bitSizeOf(u64)) |i| {
450 if ((elem & (@as(u64, 1) << @intCast(i))) != 0) global.add(sli, want_file);
451 sli += 1;
452 }
453 }
454 return Slice(SourceLocationIndex).init(global.result.items);
455}
lib/std/Build.zig+3
......@@ -22,6 +22,8 @@ pub const Step = @import("Build/Step.zig");
2222pub const Module = @import("Build/Module.zig");
2323pub const Watch = @import("Build/Watch.zig");
2424pub const Fuzz = @import("Build/Fuzz.zig");
25pub const WebServer = @import("Build/WebServer.zig");
26pub const abi = @import("Build/abi.zig");
2527
2628/// Shared state among all Build instances.
2729graph: *Graph,
......@@ -125,6 +127,7 @@ pub const Graph = struct {
125127 random_seed: u32 = 0,
126128 dependency_cache: InitializedDepMap = .empty,
127129 allow_so_scripts: ?bool = null,
130 time_report: bool,
128131};
129132
130133const AvailableDeps = []const struct { []const u8, []const u8 };
lib/std/Build/Fuzz.zig+370-81
......@@ -1,108 +1,134 @@
1const builtin = @import("builtin");
21const std = @import("../std.zig");
32const Build = std.Build;
3const Cache = Build.Cache;
44const Step = std.Build.Step;
55const assert = std.debug.assert;
66const fatal = std.process.fatal;
77const Allocator = std.mem.Allocator;
88const log = std.log;
9const Coverage = std.debug.Coverage;
10const abi = Build.abi.fuzz;
911
1012const Fuzz = @This();
1113const build_runner = @import("root");
1214
13pub const WebServer = @import("Fuzz/WebServer.zig");
14pub const abi = @import("Fuzz/abi.zig");
15
16pub fn start(
17 gpa: Allocator,
18 arena: Allocator,
19 global_cache_directory: Build.Cache.Directory,
20 zig_lib_directory: Build.Cache.Directory,
21 zig_exe_path: []const u8,
22 thread_pool: *std.Thread.Pool,
23 all_steps: []const *Step,
24 ttyconf: std.io.tty.Config,
25 listen_address: std.net.Address,
26 prog_node: std.Progress.Node,
27) Allocator.Error!void {
28 const fuzz_run_steps = block: {
29 const rebuild_node = prog_node.start("Rebuilding Unit Tests", 0);
15ws: *Build.WebServer,
16
17/// Allocated into `ws.gpa`.
18run_steps: []const *Step.Run,
19
20wait_group: std.Thread.WaitGroup,
21prog_node: std.Progress.Node,
22
23/// Protects `coverage_files`.
24coverage_mutex: std.Thread.Mutex,
25coverage_files: std.AutoArrayHashMapUnmanaged(u64, CoverageMap),
26
27queue_mutex: std.Thread.Mutex,
28queue_cond: std.Thread.Condition,
29msg_queue: std.ArrayListUnmanaged(Msg),
30
31const Msg = union(enum) {
32 coverage: struct {
33 id: u64,
34 run: *Step.Run,
35 },
36 entry_point: struct {
37 coverage_id: u64,
38 addr: u64,
39 },
40};
41
42const CoverageMap = struct {
43 mapped_memory: []align(std.heap.page_size_min) const u8,
44 coverage: Coverage,
45 source_locations: []Coverage.SourceLocation,
46 /// Elements are indexes into `source_locations` pointing to the unit tests that are being fuzz tested.
47 entry_points: std.ArrayListUnmanaged(u32),
48 start_timestamp: i64,
49
50 fn deinit(cm: *CoverageMap, gpa: Allocator) void {
51 std.posix.munmap(cm.mapped_memory);
52 cm.coverage.deinit(gpa);
53 cm.* = undefined;
54 }
55};
56
57pub fn init(ws: *Build.WebServer) Allocator.Error!Fuzz {
58 const gpa = ws.gpa;
59
60 const run_steps: []const *Step.Run = steps: {
61 var steps: std.ArrayListUnmanaged(*Step.Run) = .empty;
62 defer steps.deinit(gpa);
63 const rebuild_node = ws.root_prog_node.start("Rebuilding Unit Tests", 0);
3064 defer rebuild_node.end();
31 var wait_group: std.Thread.WaitGroup = .{};
32 defer wait_group.wait();
33 var fuzz_run_steps: std.ArrayListUnmanaged(*Step.Run) = .empty;
34 defer fuzz_run_steps.deinit(gpa);
35 for (all_steps) |step| {
65 var rebuild_wg: std.Thread.WaitGroup = .{};
66 defer rebuild_wg.wait();
67
68 for (ws.all_steps) |step| {
3669 const run = step.cast(Step.Run) orelse continue;
37 if (run.fuzz_tests.items.len > 0 and run.producer != null) {
38 thread_pool.spawnWg(&wait_group, rebuildTestsWorkerRun, .{ run, ttyconf, rebuild_node });
39 try fuzz_run_steps.append(gpa, run);
40 }
70 if (run.producer == null) continue;
71 if (run.fuzz_tests.items.len == 0) continue;
72 try steps.append(gpa, run);
73 ws.thread_pool.spawnWg(&rebuild_wg, rebuildTestsWorkerRun, .{ run, gpa, ws.ttyconf, rebuild_node });
4174 }
42 if (fuzz_run_steps.items.len == 0) fatal("no fuzz tests found", .{});
43 rebuild_node.setEstimatedTotalItems(fuzz_run_steps.items.len);
44 break :block try arena.dupe(*Step.Run, fuzz_run_steps.items);
75
76 if (steps.items.len == 0) fatal("no fuzz tests found", .{});
77 rebuild_node.setEstimatedTotalItems(steps.items.len);
78 break :steps try gpa.dupe(*Step.Run, steps.items);
4579 };
80 errdefer gpa.free(run_steps);
4681
47 // Detect failure.
48 for (fuzz_run_steps) |run| {
82 for (run_steps) |run| {
4983 assert(run.fuzz_tests.items.len > 0);
5084 if (run.rebuilt_executable == null)
5185 fatal("one or more unit tests failed to be rebuilt in fuzz mode", .{});
5286 }
5387
54 var web_server: WebServer = .{
55 .gpa = gpa,
56 .global_cache_directory = global_cache_directory,
57 .zig_lib_directory = zig_lib_directory,
58 .zig_exe_path = zig_exe_path,
59 .listen_address = listen_address,
60 .fuzz_run_steps = fuzz_run_steps,
61
62 .msg_queue = .{},
63 .mutex = .{},
64 .condition = .{},
65
66 .coverage_files = .{},
88 return .{
89 .ws = ws,
90 .run_steps = run_steps,
91 .wait_group = .{},
92 .prog_node = .none,
93 .coverage_files = .empty,
6794 .coverage_mutex = .{},
68 .coverage_condition = .{},
69
70 .base_timestamp = std.time.nanoTimestamp(),
95 .queue_mutex = .{},
96 .queue_cond = .{},
97 .msg_queue = .empty,
7198 };
99}
72100
73 // For accepting HTTP connections.
74 const web_server_thread = std.Thread.spawn(.{}, WebServer.run, .{&web_server}) catch |err| {
75 fatal("unable to spawn web server thread: {s}", .{@errorName(err)});
76 };
77 defer web_server_thread.join();
101pub fn start(fuzz: *Fuzz) void {
102 const ws = fuzz.ws;
103 fuzz.prog_node = ws.root_prog_node.start("Fuzzing", fuzz.run_steps.len);
78104
79105 // For polling messages and sending updates to subscribers.
80 const coverage_thread = std.Thread.spawn(.{}, WebServer.coverageRun, .{&web_server}) catch |err| {
106 fuzz.wait_group.start();
107 _ = std.Thread.spawn(.{}, coverageRun, .{fuzz}) catch |err| {
108 fuzz.wait_group.finish();
81109 fatal("unable to spawn coverage thread: {s}", .{@errorName(err)});
82110 };
83 defer coverage_thread.join();
84
85 {
86 const fuzz_node = prog_node.start("Fuzzing", fuzz_run_steps.len);
87 defer fuzz_node.end();
88 var wait_group: std.Thread.WaitGroup = .{};
89 defer wait_group.wait();
90111
91 for (fuzz_run_steps) |run| {
92 for (run.fuzz_tests.items) |unit_test_index| {
93 assert(run.rebuilt_executable != null);
94 thread_pool.spawnWg(&wait_group, fuzzWorkerRun, .{
95 run, &web_server, unit_test_index, ttyconf, fuzz_node,
96 });
97 }
112 for (fuzz.run_steps) |run| {
113 for (run.fuzz_tests.items) |unit_test_index| {
114 assert(run.rebuilt_executable != null);
115 ws.thread_pool.spawnWg(&fuzz.wait_group, fuzzWorkerRun, .{
116 fuzz, run, unit_test_index,
117 });
98118 }
99119 }
120}
121pub fn deinit(fuzz: *Fuzz) void {
122 if (true) @panic("TODO: terminate the fuzzer processes");
123 fuzz.wait_group.wait();
124 fuzz.prog_node.end();
100125
101 log.err("all fuzz workers crashed", .{});
126 const gpa = fuzz.ws.gpa;
127 gpa.free(fuzz.run_steps);
102128}
103129
104fn rebuildTestsWorkerRun(run: *Step.Run, ttyconf: std.io.tty.Config, parent_prog_node: std.Progress.Node) void {
105 rebuildTestsWorkerRunFallible(run, ttyconf, parent_prog_node) catch |err| {
130fn rebuildTestsWorkerRun(run: *Step.Run, gpa: Allocator, ttyconf: std.io.tty.Config, parent_prog_node: std.Progress.Node) void {
131 rebuildTestsWorkerRunFallible(run, gpa, ttyconf, parent_prog_node) catch |err| {
106132 const compile = run.producer.?;
107133 log.err("step '{s}': failed to rebuild in fuzz mode: {s}", .{
108134 compile.step.name, @errorName(err),
......@@ -110,14 +136,12 @@ fn rebuildTestsWorkerRun(run: *Step.Run, ttyconf: std.io.tty.Config, parent_prog
110136 };
111137}
112138
113fn rebuildTestsWorkerRunFallible(run: *Step.Run, ttyconf: std.io.tty.Config, parent_prog_node: std.Progress.Node) !void {
114 const gpa = run.step.owner.allocator;
115
139fn rebuildTestsWorkerRunFallible(run: *Step.Run, gpa: Allocator, ttyconf: std.io.tty.Config, parent_prog_node: std.Progress.Node) !void {
116140 const compile = run.producer.?;
117141 const prog_node = parent_prog_node.start(compile.step.name, 0);
118142 defer prog_node.end();
119143
120 const result = compile.rebuildInFuzzMode(prog_node);
144 const result = compile.rebuildInFuzzMode(gpa, prog_node);
121145
122146 const show_compile_errors = compile.step.result_error_bundle.errorMessageCount() > 0;
123147 const show_error_msgs = compile.step.result_error_msgs.items.len > 0;
......@@ -138,24 +162,22 @@ fn rebuildTestsWorkerRunFallible(run: *Step.Run, ttyconf: std.io.tty.Config, par
138162}
139163
140164fn fuzzWorkerRun(
165 fuzz: *Fuzz,
141166 run: *Step.Run,
142 web_server: *WebServer,
143167 unit_test_index: u32,
144 ttyconf: std.io.tty.Config,
145 parent_prog_node: std.Progress.Node,
146168) void {
147169 const gpa = run.step.owner.allocator;
148170 const test_name = run.cached_test_metadata.?.testName(unit_test_index);
149171
150 const prog_node = parent_prog_node.start(test_name, 0);
172 const prog_node = fuzz.prog_node.start(test_name, 0);
151173 defer prog_node.end();
152174
153 run.rerunInFuzzMode(web_server, unit_test_index, prog_node) catch |err| switch (err) {
175 run.rerunInFuzzMode(fuzz, unit_test_index, prog_node) catch |err| switch (err) {
154176 error.MakeFailed => {
155177 var buf: [256]u8 = undefined;
156178 const w = std.debug.lockStderrWriter(&buf);
157179 defer std.debug.unlockStderrWriter();
158 build_runner.printErrorMessages(gpa, &run.step, .{ .ttyconf = ttyconf }, w, false) catch {};
180 build_runner.printErrorMessages(gpa, &run.step, .{ .ttyconf = fuzz.ws.ttyconf }, w, false) catch {};
159181 return;
160182 },
161183 else => {
......@@ -166,3 +188,270 @@ fn fuzzWorkerRun(
166188 },
167189 };
168190}
191
192pub fn serveSourcesTar(fuzz: *Fuzz, req: *std.http.Server.Request) !void {
193 const gpa = fuzz.ws.gpa;
194
195 var arena_state: std.heap.ArenaAllocator = .init(gpa);
196 defer arena_state.deinit();
197 const arena = arena_state.allocator();
198
199 const DedupTable = std.ArrayHashMapUnmanaged(Build.Cache.Path, void, Build.Cache.Path.TableAdapter, false);
200 var dedup_table: DedupTable = .empty;
201 defer dedup_table.deinit(gpa);
202
203 for (fuzz.run_steps) |run_step| {
204 const compile_inputs = run_step.producer.?.step.inputs.table;
205 for (compile_inputs.keys(), compile_inputs.values()) |dir_path, *file_list| {
206 try dedup_table.ensureUnusedCapacity(gpa, file_list.items.len);
207 for (file_list.items) |sub_path| {
208 if (!std.mem.endsWith(u8, sub_path, ".zig")) continue;
209 const joined_path = try dir_path.join(arena, sub_path);
210 dedup_table.putAssumeCapacity(joined_path, {});
211 }
212 }
213 }
214
215 const deduped_paths = dedup_table.keys();
216 const SortContext = struct {
217 pub fn lessThan(this: @This(), lhs: Build.Cache.Path, rhs: Build.Cache.Path) bool {
218 _ = this;
219 return switch (std.mem.order(u8, lhs.root_dir.path orelse ".", rhs.root_dir.path orelse ".")) {
220 .lt => true,
221 .gt => false,
222 .eq => std.mem.lessThan(u8, lhs.sub_path, rhs.sub_path),
223 };
224 }
225 };
226 std.mem.sortUnstable(Build.Cache.Path, deduped_paths, SortContext{}, SortContext.lessThan);
227 return fuzz.ws.serveTarFile(req, deduped_paths);
228}
229
230pub const Previous = struct {
231 unique_runs: usize,
232 entry_points: usize,
233 pub const init: Previous = .{ .unique_runs = 0, .entry_points = 0 };
234};
235pub fn sendUpdate(
236 fuzz: *Fuzz,
237 socket: *std.http.WebSocket,
238 prev: *Previous,
239) !void {
240 fuzz.coverage_mutex.lock();
241 defer fuzz.coverage_mutex.unlock();
242
243 const coverage_maps = fuzz.coverage_files.values();
244 if (coverage_maps.len == 0) return;
245 // TODO: handle multiple fuzz steps in the WebSocket packets
246 const coverage_map = &coverage_maps[0];
247 const cov_header: *const abi.SeenPcsHeader = @ptrCast(coverage_map.mapped_memory[0..@sizeOf(abi.SeenPcsHeader)]);
248 // TODO: this isn't sound! We need to do volatile reads of these bits rather than handing the
249 // buffer off to the kernel, because we might race with the fuzzer process[es]. This brings the
250 // whole mmap strategy into question. Incidentally, I wonder if post-writergate we could pass
251 // this data straight to the socket with sendfile...
252 const seen_pcs = cov_header.seenBits();
253 const n_runs = @atomicLoad(usize, &cov_header.n_runs, .monotonic);
254 const unique_runs = @atomicLoad(usize, &cov_header.unique_runs, .monotonic);
255 if (prev.unique_runs != unique_runs) {
256 // There has been an update.
257 if (prev.unique_runs == 0) {
258 // We need to send initial context.
259 const header: abi.SourceIndexHeader = .{
260 .directories_len = @intCast(coverage_map.coverage.directories.entries.len),
261 .files_len = @intCast(coverage_map.coverage.files.entries.len),
262 .source_locations_len = @intCast(coverage_map.source_locations.len),
263 .string_bytes_len = @intCast(coverage_map.coverage.string_bytes.items.len),
264 .start_timestamp = coverage_map.start_timestamp,
265 };
266 const iovecs: [5]std.posix.iovec_const = .{
267 makeIov(@ptrCast(&header)),
268 makeIov(@ptrCast(coverage_map.coverage.directories.keys())),
269 makeIov(@ptrCast(coverage_map.coverage.files.keys())),
270 makeIov(@ptrCast(coverage_map.source_locations)),
271 makeIov(coverage_map.coverage.string_bytes.items),
272 };
273 try socket.writeMessagev(&iovecs, .binary);
274 }
275
276 const header: abi.CoverageUpdateHeader = .{
277 .n_runs = n_runs,
278 .unique_runs = unique_runs,
279 };
280 const iovecs: [2]std.posix.iovec_const = .{
281 makeIov(@ptrCast(&header)),
282 makeIov(@ptrCast(seen_pcs)),
283 };
284 try socket.writeMessagev(&iovecs, .binary);
285
286 prev.unique_runs = unique_runs;
287 }
288
289 if (prev.entry_points != coverage_map.entry_points.items.len) {
290 const header: abi.EntryPointHeader = .init(@intCast(coverage_map.entry_points.items.len));
291 const iovecs: [2]std.posix.iovec_const = .{
292 makeIov(@ptrCast(&header)),
293 makeIov(@ptrCast(coverage_map.entry_points.items)),
294 };
295 try socket.writeMessagev(&iovecs, .binary);
296
297 prev.entry_points = coverage_map.entry_points.items.len;
298 }
299}
300
301fn coverageRun(fuzz: *Fuzz) void {
302 defer fuzz.wait_group.finish();
303
304 fuzz.queue_mutex.lock();
305 defer fuzz.queue_mutex.unlock();
306
307 while (true) {
308 fuzz.queue_cond.wait(&fuzz.queue_mutex);
309 for (fuzz.msg_queue.items) |msg| switch (msg) {
310 .coverage => |coverage| prepareTables(fuzz, coverage.run, coverage.id) catch |err| switch (err) {
311 error.AlreadyReported => continue,
312 else => |e| log.err("failed to prepare code coverage tables: {s}", .{@errorName(e)}),
313 },
314 .entry_point => |entry_point| addEntryPoint(fuzz, entry_point.coverage_id, entry_point.addr) catch |err| switch (err) {
315 error.AlreadyReported => continue,
316 else => |e| log.err("failed to prepare code coverage tables: {s}", .{@errorName(e)}),
317 },
318 };
319 fuzz.msg_queue.clearRetainingCapacity();
320 }
321}
322fn prepareTables(fuzz: *Fuzz, run_step: *Step.Run, coverage_id: u64) error{ OutOfMemory, AlreadyReported }!void {
323 const ws = fuzz.ws;
324 const gpa = ws.gpa;
325
326 fuzz.coverage_mutex.lock();
327 defer fuzz.coverage_mutex.unlock();
328
329 const gop = try fuzz.coverage_files.getOrPut(gpa, coverage_id);
330 if (gop.found_existing) {
331 // We are fuzzing the same executable with multiple threads.
332 // Perhaps the same unit test; perhaps a different one. In any
333 // case, since the coverage file is the same, we only have to
334 // notice changes to that one file in order to learn coverage for
335 // this particular executable.
336 return;
337 }
338 errdefer _ = fuzz.coverage_files.pop();
339
340 gop.value_ptr.* = .{
341 .coverage = std.debug.Coverage.init,
342 .mapped_memory = undefined, // populated below
343 .source_locations = undefined, // populated below
344 .entry_points = .{},
345 .start_timestamp = ws.now(),
346 };
347 errdefer gop.value_ptr.coverage.deinit(gpa);
348
349 const rebuilt_exe_path = run_step.rebuilt_executable.?;
350 var debug_info = std.debug.Info.load(gpa, rebuilt_exe_path, &gop.value_ptr.coverage) catch |err| {
351 log.err("step '{s}': failed to load debug information for '{f}': {s}", .{
352 run_step.step.name, rebuilt_exe_path, @errorName(err),
353 });
354 return error.AlreadyReported;
355 };
356 defer debug_info.deinit(gpa);
357
358 const coverage_file_path: Build.Cache.Path = .{
359 .root_dir = run_step.step.owner.cache_root,
360 .sub_path = "v/" ++ std.fmt.hex(coverage_id),
361 };
362 var coverage_file = coverage_file_path.root_dir.handle.openFile(coverage_file_path.sub_path, .{}) catch |err| {
363 log.err("step '{s}': failed to load coverage file '{f}': {s}", .{
364 run_step.step.name, coverage_file_path, @errorName(err),
365 });
366 return error.AlreadyReported;
367 };
368 defer coverage_file.close();
369
370 const file_size = coverage_file.getEndPos() catch |err| {
371 log.err("unable to check len of coverage file '{f}': {s}", .{ coverage_file_path, @errorName(err) });
372 return error.AlreadyReported;
373 };
374
375 const mapped_memory = std.posix.mmap(
376 null,
377 file_size,
378 std.posix.PROT.READ,
379 .{ .TYPE = .SHARED },
380 coverage_file.handle,
381 0,
382 ) catch |err| {
383 log.err("failed to map coverage file '{f}': {s}", .{ coverage_file_path, @errorName(err) });
384 return error.AlreadyReported;
385 };
386 gop.value_ptr.mapped_memory = mapped_memory;
387
388 const header: *const abi.SeenPcsHeader = @ptrCast(mapped_memory[0..@sizeOf(abi.SeenPcsHeader)]);
389 const pcs = header.pcAddrs();
390 const source_locations = try gpa.alloc(Coverage.SourceLocation, pcs.len);
391 errdefer gpa.free(source_locations);
392
393 // Unfortunately the PCs array that LLVM gives us from the 8-bit PC
394 // counters feature is not sorted.
395 var sorted_pcs: std.MultiArrayList(struct { pc: u64, index: u32, sl: Coverage.SourceLocation }) = .{};
396 defer sorted_pcs.deinit(gpa);
397 try sorted_pcs.resize(gpa, pcs.len);
398 @memcpy(sorted_pcs.items(.pc), pcs);
399 for (sorted_pcs.items(.index), 0..) |*v, i| v.* = @intCast(i);
400 sorted_pcs.sortUnstable(struct {
401 addrs: []const u64,
402
403 pub fn lessThan(ctx: @This(), a_index: usize, b_index: usize) bool {
404 return ctx.addrs[a_index] < ctx.addrs[b_index];
405 }
406 }{ .addrs = sorted_pcs.items(.pc) });
407
408 debug_info.resolveAddresses(gpa, sorted_pcs.items(.pc), sorted_pcs.items(.sl)) catch |err| {
409 log.err("failed to resolve addresses to source locations: {s}", .{@errorName(err)});
410 return error.AlreadyReported;
411 };
412
413 for (sorted_pcs.items(.index), sorted_pcs.items(.sl)) |i, sl| source_locations[i] = sl;
414 gop.value_ptr.source_locations = source_locations;
415
416 ws.notifyUpdate();
417}
418fn addEntryPoint(fuzz: *Fuzz, coverage_id: u64, addr: u64) error{ AlreadyReported, OutOfMemory }!void {
419 fuzz.coverage_mutex.lock();
420 defer fuzz.coverage_mutex.unlock();
421
422 const coverage_map = fuzz.coverage_files.getPtr(coverage_id).?;
423 const header: *const abi.SeenPcsHeader = @ptrCast(coverage_map.mapped_memory[0..@sizeOf(abi.SeenPcsHeader)]);
424 const pcs = header.pcAddrs();
425
426 // Since this pcs list is unsorted, we must linear scan for the best index.
427 const index = i: {
428 var best: usize = 0;
429 for (pcs[1..], 1..) |elem_addr, i| {
430 if (elem_addr == addr) break :i i;
431 if (elem_addr > addr) continue;
432 if (elem_addr > pcs[best]) best = i;
433 }
434 break :i best;
435 };
436 if (index >= pcs.len) {
437 log.err("unable to find unit test entry address 0x{x} in source locations (range: 0x{x} to 0x{x})", .{
438 addr, pcs[0], pcs[pcs.len - 1],
439 });
440 return error.AlreadyReported;
441 }
442 if (false) {
443 const sl = coverage_map.source_locations[index];
444 const file_name = coverage_map.coverage.stringAt(coverage_map.coverage.fileAt(sl.file).basename);
445 log.debug("server found entry point for 0x{x} at {s}:{d}:{d} - index {d} between {x} and {x}", .{
446 addr, file_name, sl.line, sl.column, index, pcs[index - 1], pcs[index + 1],
447 });
448 }
449 try coverage_map.entry_points.append(fuzz.ws.gpa, @intCast(index));
450}
451
452fn makeIov(s: []const u8) std.posix.iovec_const {
453 return .{
454 .base = s.ptr,
455 .len = s.len,
456 };
457}
lib/std/Build/Fuzz/WebServer.zig deleted-709
......@@ -1,709 +0,0 @@
1const builtin = @import("builtin");
2
3const std = @import("../../std.zig");
4const Allocator = std.mem.Allocator;
5const Build = std.Build;
6const Step = std.Build.Step;
7const Coverage = std.debug.Coverage;
8const abi = std.Build.Fuzz.abi;
9const log = std.log;
10const assert = std.debug.assert;
11const Cache = std.Build.Cache;
12const Path = Cache.Path;
13
14const WebServer = @This();
15
16gpa: Allocator,
17global_cache_directory: Build.Cache.Directory,
18zig_lib_directory: Build.Cache.Directory,
19zig_exe_path: []const u8,
20listen_address: std.net.Address,
21fuzz_run_steps: []const *Step.Run,
22
23/// Messages from fuzz workers. Protected by mutex.
24msg_queue: std.ArrayListUnmanaged(Msg),
25/// Protects `msg_queue` only.
26mutex: std.Thread.Mutex,
27/// Signaled when there is a message in `msg_queue`.
28condition: std.Thread.Condition,
29
30coverage_files: std.AutoArrayHashMapUnmanaged(u64, CoverageMap),
31/// Protects `coverage_files` only.
32coverage_mutex: std.Thread.Mutex,
33/// Signaled when `coverage_files` changes.
34coverage_condition: std.Thread.Condition,
35
36/// Time at initialization of WebServer.
37base_timestamp: i128,
38
39const fuzzer_bin_name = "fuzzer";
40const fuzzer_arch_os_abi = "wasm32-freestanding";
41const fuzzer_cpu_features = "baseline+atomics+bulk_memory+multivalue+mutable_globals+nontrapping_fptoint+reference_types+sign_ext";
42
43const CoverageMap = struct {
44 mapped_memory: []align(std.heap.page_size_min) const u8,
45 coverage: Coverage,
46 source_locations: []Coverage.SourceLocation,
47 /// Elements are indexes into `source_locations` pointing to the unit tests that are being fuzz tested.
48 entry_points: std.ArrayListUnmanaged(u32),
49 start_timestamp: i64,
50
51 fn deinit(cm: *CoverageMap, gpa: Allocator) void {
52 std.posix.munmap(cm.mapped_memory);
53 cm.coverage.deinit(gpa);
54 cm.* = undefined;
55 }
56};
57
58const Msg = union(enum) {
59 coverage: struct {
60 id: u64,
61 run: *Step.Run,
62 },
63 entry_point: struct {
64 coverage_id: u64,
65 addr: u64,
66 },
67};
68
69pub fn run(ws: *WebServer) void {
70 var http_server = ws.listen_address.listen(.{
71 .reuse_address = true,
72 }) catch |err| {
73 log.err("failed to listen to port {d}: {s}", .{ ws.listen_address.in.getPort(), @errorName(err) });
74 return;
75 };
76 const port = http_server.listen_address.in.getPort();
77 log.info("web interface listening at http://127.0.0.1:{d}/", .{port});
78 if (ws.listen_address.in.getPort() == 0)
79 log.info("hint: pass --port {d} to use this same port next time", .{port});
80
81 while (true) {
82 const connection = http_server.accept() catch |err| {
83 log.err("failed to accept connection: {s}", .{@errorName(err)});
84 return;
85 };
86 _ = std.Thread.spawn(.{}, accept, .{ ws, connection }) catch |err| {
87 log.err("unable to spawn connection thread: {s}", .{@errorName(err)});
88 connection.stream.close();
89 continue;
90 };
91 }
92}
93
94fn now(s: *const WebServer) i64 {
95 return @intCast(std.time.nanoTimestamp() - s.base_timestamp);
96}
97
98fn accept(ws: *WebServer, connection: std.net.Server.Connection) void {
99 defer connection.stream.close();
100
101 var read_buffer: [0x4000]u8 = undefined;
102 var server = std.http.Server.init(connection, &read_buffer);
103 var web_socket: std.http.WebSocket = undefined;
104 var send_buffer: [0x4000]u8 = undefined;
105 var ws_recv_buffer: [0x4000]u8 align(4) = undefined;
106 while (server.state == .ready) {
107 var request = server.receiveHead() catch |err| switch (err) {
108 error.HttpConnectionClosing => return,
109 else => {
110 log.err("closing http connection: {s}", .{@errorName(err)});
111 return;
112 },
113 };
114 if (web_socket.init(&request, &send_buffer, &ws_recv_buffer) catch |err| {
115 log.err("initializing web socket: {s}", .{@errorName(err)});
116 return;
117 }) {
118 serveWebSocket(ws, &web_socket) catch |err| {
119 log.err("unable to serve web socket connection: {s}", .{@errorName(err)});
120 return;
121 };
122 } else {
123 serveRequest(ws, &request) catch |err| switch (err) {
124 error.AlreadyReported => return,
125 else => |e| {
126 log.err("unable to serve {s}: {s}", .{ request.head.target, @errorName(e) });
127 return;
128 },
129 };
130 }
131 }
132}
133
134fn serveRequest(ws: *WebServer, request: *std.http.Server.Request) !void {
135 if (std.mem.eql(u8, request.head.target, "/") or
136 std.mem.eql(u8, request.head.target, "/debug") or
137 std.mem.eql(u8, request.head.target, "/debug/"))
138 {
139 try serveFile(ws, request, "fuzzer/web/index.html", "text/html");
140 } else if (std.mem.eql(u8, request.head.target, "/main.js") or
141 std.mem.eql(u8, request.head.target, "/debug/main.js"))
142 {
143 try serveFile(ws, request, "fuzzer/web/main.js", "application/javascript");
144 } else if (std.mem.eql(u8, request.head.target, "/main.wasm")) {
145 try serveWasm(ws, request, .ReleaseFast);
146 } else if (std.mem.eql(u8, request.head.target, "/debug/main.wasm")) {
147 try serveWasm(ws, request, .Debug);
148 } else if (std.mem.eql(u8, request.head.target, "/sources.tar") or
149 std.mem.eql(u8, request.head.target, "/debug/sources.tar"))
150 {
151 try serveSourcesTar(ws, request);
152 } else {
153 try request.respond("not found", .{
154 .status = .not_found,
155 .extra_headers = &.{
156 .{ .name = "content-type", .value = "text/plain" },
157 },
158 });
159 }
160}
161
162fn serveFile(
163 ws: *WebServer,
164 request: *std.http.Server.Request,
165 name: []const u8,
166 content_type: []const u8,
167) !void {
168 const gpa = ws.gpa;
169 // The desired API is actually sendfile, which will require enhancing std.http.Server.
170 // We load the file with every request so that the user can make changes to the file
171 // and refresh the HTML page without restarting this server.
172 const file_contents = ws.zig_lib_directory.handle.readFileAlloc(gpa, name, 10 * 1024 * 1024) catch |err| {
173 log.err("failed to read '{f}{s}': {s}", .{ ws.zig_lib_directory, name, @errorName(err) });
174 return error.AlreadyReported;
175 };
176 defer gpa.free(file_contents);
177 try request.respond(file_contents, .{
178 .extra_headers = &.{
179 .{ .name = "content-type", .value = content_type },
180 cache_control_header,
181 },
182 });
183}
184
185fn serveWasm(
186 ws: *WebServer,
187 request: *std.http.Server.Request,
188 optimize_mode: std.builtin.OptimizeMode,
189) !void {
190 const gpa = ws.gpa;
191
192 var arena_instance = std.heap.ArenaAllocator.init(gpa);
193 defer arena_instance.deinit();
194 const arena = arena_instance.allocator();
195
196 // Do the compilation every request, so that the user can edit the files
197 // and see the changes without restarting the server.
198 const wasm_base_path = try buildWasmBinary(ws, arena, optimize_mode);
199 const bin_name = try std.zig.binNameAlloc(arena, .{
200 .root_name = fuzzer_bin_name,
201 .target = &(std.zig.system.resolveTargetQuery(std.Build.parseTargetQuery(.{
202 .arch_os_abi = fuzzer_arch_os_abi,
203 .cpu_features = fuzzer_cpu_features,
204 }) catch unreachable) catch unreachable),
205 .output_mode = .Exe,
206 });
207 // std.http.Server does not have a sendfile API yet.
208 const bin_path = try wasm_base_path.join(arena, bin_name);
209 const file_contents = try bin_path.root_dir.handle.readFileAlloc(gpa, bin_path.sub_path, 10 * 1024 * 1024);
210 defer gpa.free(file_contents);
211 try request.respond(file_contents, .{
212 .extra_headers = &.{
213 .{ .name = "content-type", .value = "application/wasm" },
214 cache_control_header,
215 },
216 });
217}
218
219fn buildWasmBinary(
220 ws: *WebServer,
221 arena: Allocator,
222 optimize_mode: std.builtin.OptimizeMode,
223) !Path {
224 const gpa = ws.gpa;
225
226 const main_src_path: Build.Cache.Path = .{
227 .root_dir = ws.zig_lib_directory,
228 .sub_path = "fuzzer/web/main.zig",
229 };
230 const walk_src_path: Build.Cache.Path = .{
231 .root_dir = ws.zig_lib_directory,
232 .sub_path = "docs/wasm/Walk.zig",
233 };
234 const html_render_src_path: Build.Cache.Path = .{
235 .root_dir = ws.zig_lib_directory,
236 .sub_path = "docs/wasm/html_render.zig",
237 };
238
239 var argv: std.ArrayListUnmanaged([]const u8) = .empty;
240
241 try argv.appendSlice(arena, &.{
242 ws.zig_exe_path, "build-exe", //
243 "-fno-entry", //
244 "-O", @tagName(optimize_mode), //
245 "-target", fuzzer_arch_os_abi, //
246 "-mcpu", fuzzer_cpu_features, //
247 "--cache-dir", ws.global_cache_directory.path orelse ".", //
248 "--global-cache-dir", ws.global_cache_directory.path orelse ".", //
249 "--name", fuzzer_bin_name, //
250 "-rdynamic", //
251 "-fsingle-threaded", //
252 "--dep", "Walk", //
253 "--dep", "html_render", //
254 try std.fmt.allocPrint(arena, "-Mroot={f}", .{main_src_path}), //
255 try std.fmt.allocPrint(arena, "-MWalk={f}", .{walk_src_path}), //
256 "--dep", "Walk", //
257 try std.fmt.allocPrint(arena, "-Mhtml_render={f}", .{html_render_src_path}), //
258 "--listen=-",
259 });
260
261 var child = std.process.Child.init(argv.items, gpa);
262 child.stdin_behavior = .Pipe;
263 child.stdout_behavior = .Pipe;
264 child.stderr_behavior = .Pipe;
265 try child.spawn();
266
267 var poller = std.io.poll(gpa, enum { stdout, stderr }, .{
268 .stdout = child.stdout.?,
269 .stderr = child.stderr.?,
270 });
271 defer poller.deinit();
272
273 try sendMessage(child.stdin.?, .update);
274 try sendMessage(child.stdin.?, .exit);
275
276 var result: ?Path = null;
277 var result_error_bundle = std.zig.ErrorBundle.empty;
278
279 const stdout = poller.reader(.stdout);
280
281 poll: while (true) {
282 const Header = std.zig.Server.Message.Header;
283 while (stdout.buffered().len < @sizeOf(Header)) if (!try poller.poll()) break :poll;
284 const header = stdout.takeStruct(Header, .little) catch unreachable;
285 while (stdout.buffered().len < header.bytes_len) if (!try poller.poll()) break :poll;
286 const body = stdout.take(header.bytes_len) catch unreachable;
287
288 switch (header.tag) {
289 .zig_version => {
290 if (!std.mem.eql(u8, builtin.zig_version_string, body)) {
291 return error.ZigProtocolVersionMismatch;
292 }
293 },
294 .error_bundle => {
295 const EbHdr = std.zig.Server.Message.ErrorBundle;
296 const eb_hdr = @as(*align(1) const EbHdr, @ptrCast(body));
297 const extra_bytes =
298 body[@sizeOf(EbHdr)..][0 .. @sizeOf(u32) * eb_hdr.extra_len];
299 const string_bytes =
300 body[@sizeOf(EbHdr) + extra_bytes.len ..][0..eb_hdr.string_bytes_len];
301 // TODO: use @ptrCast when the compiler supports it
302 const unaligned_extra = std.mem.bytesAsSlice(u32, extra_bytes);
303 const extra_array = try arena.alloc(u32, unaligned_extra.len);
304 @memcpy(extra_array, unaligned_extra);
305 result_error_bundle = .{
306 .string_bytes = try arena.dupe(u8, string_bytes),
307 .extra = extra_array,
308 };
309 },
310 .emit_digest => {
311 const EmitDigest = std.zig.Server.Message.EmitDigest;
312 const ebp_hdr = @as(*align(1) const EmitDigest, @ptrCast(body));
313 if (!ebp_hdr.flags.cache_hit) {
314 log.info("source changes detected; rebuilt wasm component", .{});
315 }
316 const digest = body[@sizeOf(EmitDigest)..][0..Cache.bin_digest_len];
317 result = .{
318 .root_dir = ws.global_cache_directory,
319 .sub_path = try arena.dupe(u8, "o" ++ std.fs.path.sep_str ++ Cache.binToHex(digest.*)),
320 };
321 },
322 else => {}, // ignore other messages
323 }
324 }
325
326 const stderr_contents = try poller.toOwnedSlice(.stderr);
327 if (stderr_contents.len > 0) {
328 std.debug.print("{s}", .{stderr_contents});
329 }
330
331 // Send EOF to stdin.
332 child.stdin.?.close();
333 child.stdin = null;
334
335 switch (try child.wait()) {
336 .Exited => |code| {
337 if (code != 0) {
338 log.err(
339 "the following command exited with error code {d}:\n{s}",
340 .{ code, try Build.Step.allocPrintCmd(arena, null, argv.items) },
341 );
342 return error.WasmCompilationFailed;
343 }
344 },
345 .Signal, .Stopped, .Unknown => {
346 log.err(
347 "the following command terminated unexpectedly:\n{s}",
348 .{try Build.Step.allocPrintCmd(arena, null, argv.items)},
349 );
350 return error.WasmCompilationFailed;
351 },
352 }
353
354 if (result_error_bundle.errorMessageCount() > 0) {
355 const color = std.zig.Color.auto;
356 result_error_bundle.renderToStdErr(color.renderOptions());
357 log.err("the following command failed with {d} compilation errors:\n{s}", .{
358 result_error_bundle.errorMessageCount(),
359 try Build.Step.allocPrintCmd(arena, null, argv.items),
360 });
361 return error.WasmCompilationFailed;
362 }
363
364 return result orelse {
365 log.err("child process failed to report result\n{s}", .{
366 try Build.Step.allocPrintCmd(arena, null, argv.items),
367 });
368 return error.WasmCompilationFailed;
369 };
370}
371
372fn sendMessage(file: std.fs.File, tag: std.zig.Client.Message.Tag) !void {
373 const header: std.zig.Client.Message.Header = .{
374 .tag = tag,
375 .bytes_len = 0,
376 };
377 try file.writeAll(std.mem.asBytes(&header));
378}
379
380fn serveWebSocket(ws: *WebServer, web_socket: *std.http.WebSocket) !void {
381 ws.coverage_mutex.lock();
382 defer ws.coverage_mutex.unlock();
383
384 // On first connection, the client needs to know what time the server
385 // thinks it is to rebase timestamps.
386 {
387 const timestamp_message: abi.CurrentTime = .{ .base = ws.now() };
388 try web_socket.writeMessage(std.mem.asBytes(&timestamp_message), .binary);
389 }
390
391 // On first connection, the client needs all the coverage information
392 // so that subsequent updates can contain only the updated bits.
393 var prev_unique_runs: usize = 0;
394 var prev_entry_points: usize = 0;
395 try sendCoverageContext(ws, web_socket, &prev_unique_runs, &prev_entry_points);
396 while (true) {
397 ws.coverage_condition.timedWait(&ws.coverage_mutex, std.time.ns_per_ms * 500) catch {};
398 try sendCoverageContext(ws, web_socket, &prev_unique_runs, &prev_entry_points);
399 }
400}
401
402fn sendCoverageContext(
403 ws: *WebServer,
404 web_socket: *std.http.WebSocket,
405 prev_unique_runs: *usize,
406 prev_entry_points: *usize,
407) !void {
408 const coverage_maps = ws.coverage_files.values();
409 if (coverage_maps.len == 0) return;
410 // TODO: make each events URL correspond to one coverage map
411 const coverage_map = &coverage_maps[0];
412 const cov_header: *const abi.SeenPcsHeader = @ptrCast(coverage_map.mapped_memory[0..@sizeOf(abi.SeenPcsHeader)]);
413 const seen_pcs = cov_header.seenBits();
414 const n_runs = @atomicLoad(usize, &cov_header.n_runs, .monotonic);
415 const unique_runs = @atomicLoad(usize, &cov_header.unique_runs, .monotonic);
416 if (prev_unique_runs.* != unique_runs) {
417 // There has been an update.
418 if (prev_unique_runs.* == 0) {
419 // We need to send initial context.
420 const header: abi.SourceIndexHeader = .{
421 .flags = .{},
422 .directories_len = @intCast(coverage_map.coverage.directories.entries.len),
423 .files_len = @intCast(coverage_map.coverage.files.entries.len),
424 .source_locations_len = @intCast(coverage_map.source_locations.len),
425 .string_bytes_len = @intCast(coverage_map.coverage.string_bytes.items.len),
426 .start_timestamp = coverage_map.start_timestamp,
427 };
428 const iovecs: [5]std.posix.iovec_const = .{
429 makeIov(std.mem.asBytes(&header)),
430 makeIov(std.mem.sliceAsBytes(coverage_map.coverage.directories.keys())),
431 makeIov(std.mem.sliceAsBytes(coverage_map.coverage.files.keys())),
432 makeIov(std.mem.sliceAsBytes(coverage_map.source_locations)),
433 makeIov(coverage_map.coverage.string_bytes.items),
434 };
435 try web_socket.writeMessagev(&iovecs, .binary);
436 }
437
438 const header: abi.CoverageUpdateHeader = .{
439 .n_runs = n_runs,
440 .unique_runs = unique_runs,
441 };
442 const iovecs: [2]std.posix.iovec_const = .{
443 makeIov(std.mem.asBytes(&header)),
444 makeIov(std.mem.sliceAsBytes(seen_pcs)),
445 };
446 try web_socket.writeMessagev(&iovecs, .binary);
447
448 prev_unique_runs.* = unique_runs;
449 }
450
451 if (prev_entry_points.* != coverage_map.entry_points.items.len) {
452 const header: abi.EntryPointHeader = .{
453 .flags = .{
454 .locs_len = @intCast(coverage_map.entry_points.items.len),
455 },
456 };
457 const iovecs: [2]std.posix.iovec_const = .{
458 makeIov(std.mem.asBytes(&header)),
459 makeIov(std.mem.sliceAsBytes(coverage_map.entry_points.items)),
460 };
461 try web_socket.writeMessagev(&iovecs, .binary);
462
463 prev_entry_points.* = coverage_map.entry_points.items.len;
464 }
465}
466
467fn serveSourcesTar(ws: *WebServer, request: *std.http.Server.Request) !void {
468 const gpa = ws.gpa;
469
470 var arena_instance = std.heap.ArenaAllocator.init(gpa);
471 defer arena_instance.deinit();
472 const arena = arena_instance.allocator();
473
474 var send_buffer: [0x4000]u8 = undefined;
475 var response = request.respondStreaming(.{
476 .send_buffer = &send_buffer,
477 .respond_options = .{
478 .extra_headers = &.{
479 .{ .name = "content-type", .value = "application/x-tar" },
480 cache_control_header,
481 },
482 },
483 });
484
485 const DedupeTable = std.ArrayHashMapUnmanaged(Build.Cache.Path, void, Build.Cache.Path.TableAdapter, false);
486 var dedupe_table: DedupeTable = .{};
487 defer dedupe_table.deinit(gpa);
488
489 for (ws.fuzz_run_steps) |run_step| {
490 const compile_step_inputs = run_step.producer.?.step.inputs.table;
491 for (compile_step_inputs.keys(), compile_step_inputs.values()) |dir_path, *file_list| {
492 try dedupe_table.ensureUnusedCapacity(gpa, file_list.items.len);
493 for (file_list.items) |sub_path| {
494 // Special file "." means the entire directory.
495 if (std.mem.eql(u8, sub_path, ".")) continue;
496 const joined_path = try dir_path.join(arena, sub_path);
497 _ = dedupe_table.getOrPutAssumeCapacity(joined_path);
498 }
499 }
500 }
501
502 const deduped_paths = dedupe_table.keys();
503 const SortContext = struct {
504 pub fn lessThan(this: @This(), lhs: Build.Cache.Path, rhs: Build.Cache.Path) bool {
505 _ = this;
506 return switch (std.mem.order(u8, lhs.root_dir.path orelse ".", rhs.root_dir.path orelse ".")) {
507 .lt => true,
508 .gt => false,
509 .eq => std.mem.lessThan(u8, lhs.sub_path, rhs.sub_path),
510 };
511 }
512 };
513 std.mem.sortUnstable(Build.Cache.Path, deduped_paths, SortContext{}, SortContext.lessThan);
514
515 var cwd_cache: ?[]const u8 = null;
516
517 var adapter = response.writer().adaptToNewApi();
518 var archiver: std.tar.Writer = .{ .underlying_writer = &adapter.new_interface };
519 var read_buffer: [1024]u8 = undefined;
520
521 for (deduped_paths) |joined_path| {
522 var file = joined_path.root_dir.handle.openFile(joined_path.sub_path, .{}) catch |err| {
523 log.err("failed to open {f}: {s}", .{ joined_path, @errorName(err) });
524 continue;
525 };
526 defer file.close();
527 const stat = try file.stat();
528 var file_reader: std.fs.File.Reader = .initSize(file, &read_buffer, stat.size);
529 archiver.prefix = joined_path.root_dir.path orelse try memoizedCwd(arena, &cwd_cache);
530 try archiver.writeFile(joined_path.sub_path, &file_reader, stat.mtime);
531 }
532
533 // intentionally not calling `archiver.finishPedantically`
534 try adapter.new_interface.flush();
535 try response.end();
536}
537
538fn memoizedCwd(arena: Allocator, opt_ptr: *?[]const u8) ![]const u8 {
539 if (opt_ptr.*) |cached| return cached;
540 const result = try std.process.getCwdAlloc(arena);
541 opt_ptr.* = result;
542 return result;
543}
544
545const cache_control_header: std.http.Header = .{
546 .name = "cache-control",
547 .value = "max-age=0, must-revalidate",
548};
549
550pub fn coverageRun(ws: *WebServer) void {
551 ws.mutex.lock();
552 defer ws.mutex.unlock();
553
554 while (true) {
555 ws.condition.wait(&ws.mutex);
556 for (ws.msg_queue.items) |msg| switch (msg) {
557 .coverage => |coverage| prepareTables(ws, coverage.run, coverage.id) catch |err| switch (err) {
558 error.AlreadyReported => continue,
559 else => |e| log.err("failed to prepare code coverage tables: {s}", .{@errorName(e)}),
560 },
561 .entry_point => |entry_point| addEntryPoint(ws, entry_point.coverage_id, entry_point.addr) catch |err| switch (err) {
562 error.AlreadyReported => continue,
563 else => |e| log.err("failed to prepare code coverage tables: {s}", .{@errorName(e)}),
564 },
565 };
566 ws.msg_queue.clearRetainingCapacity();
567 }
568}
569
570fn prepareTables(
571 ws: *WebServer,
572 run_step: *Step.Run,
573 coverage_id: u64,
574) error{ OutOfMemory, AlreadyReported }!void {
575 const gpa = ws.gpa;
576
577 ws.coverage_mutex.lock();
578 defer ws.coverage_mutex.unlock();
579
580 const gop = try ws.coverage_files.getOrPut(gpa, coverage_id);
581 if (gop.found_existing) {
582 // We are fuzzing the same executable with multiple threads.
583 // Perhaps the same unit test; perhaps a different one. In any
584 // case, since the coverage file is the same, we only have to
585 // notice changes to that one file in order to learn coverage for
586 // this particular executable.
587 return;
588 }
589 errdefer _ = ws.coverage_files.pop();
590
591 gop.value_ptr.* = .{
592 .coverage = std.debug.Coverage.init,
593 .mapped_memory = undefined, // populated below
594 .source_locations = undefined, // populated below
595 .entry_points = .{},
596 .start_timestamp = ws.now(),
597 };
598 errdefer gop.value_ptr.coverage.deinit(gpa);
599
600 const rebuilt_exe_path = run_step.rebuilt_executable.?;
601 var debug_info = std.debug.Info.load(gpa, rebuilt_exe_path, &gop.value_ptr.coverage) catch |err| {
602 log.err("step '{s}': failed to load debug information for '{f}': {s}", .{
603 run_step.step.name, rebuilt_exe_path, @errorName(err),
604 });
605 return error.AlreadyReported;
606 };
607 defer debug_info.deinit(gpa);
608
609 const coverage_file_path: Build.Cache.Path = .{
610 .root_dir = run_step.step.owner.cache_root,
611 .sub_path = "v/" ++ std.fmt.hex(coverage_id),
612 };
613 var coverage_file = coverage_file_path.root_dir.handle.openFile(coverage_file_path.sub_path, .{}) catch |err| {
614 log.err("step '{s}': failed to load coverage file '{f}': {s}", .{
615 run_step.step.name, coverage_file_path, @errorName(err),
616 });
617 return error.AlreadyReported;
618 };
619 defer coverage_file.close();
620
621 const file_size = coverage_file.getEndPos() catch |err| {
622 log.err("unable to check len of coverage file '{f}': {s}", .{ coverage_file_path, @errorName(err) });
623 return error.AlreadyReported;
624 };
625
626 const mapped_memory = std.posix.mmap(
627 null,
628 file_size,
629 std.posix.PROT.READ,
630 .{ .TYPE = .SHARED },
631 coverage_file.handle,
632 0,
633 ) catch |err| {
634 log.err("failed to map coverage file '{f}': {s}", .{ coverage_file_path, @errorName(err) });
635 return error.AlreadyReported;
636 };
637 gop.value_ptr.mapped_memory = mapped_memory;
638
639 const header: *const abi.SeenPcsHeader = @ptrCast(mapped_memory[0..@sizeOf(abi.SeenPcsHeader)]);
640 const pcs = header.pcAddrs();
641 const source_locations = try gpa.alloc(Coverage.SourceLocation, pcs.len);
642 errdefer gpa.free(source_locations);
643
644 // Unfortunately the PCs array that LLVM gives us from the 8-bit PC
645 // counters feature is not sorted.
646 var sorted_pcs: std.MultiArrayList(struct { pc: u64, index: u32, sl: Coverage.SourceLocation }) = .{};
647 defer sorted_pcs.deinit(gpa);
648 try sorted_pcs.resize(gpa, pcs.len);
649 @memcpy(sorted_pcs.items(.pc), pcs);
650 for (sorted_pcs.items(.index), 0..) |*v, i| v.* = @intCast(i);
651 sorted_pcs.sortUnstable(struct {
652 addrs: []const u64,
653
654 pub fn lessThan(ctx: @This(), a_index: usize, b_index: usize) bool {
655 return ctx.addrs[a_index] < ctx.addrs[b_index];
656 }
657 }{ .addrs = sorted_pcs.items(.pc) });
658
659 debug_info.resolveAddresses(gpa, sorted_pcs.items(.pc), sorted_pcs.items(.sl)) catch |err| {
660 log.err("failed to resolve addresses to source locations: {s}", .{@errorName(err)});
661 return error.AlreadyReported;
662 };
663
664 for (sorted_pcs.items(.index), sorted_pcs.items(.sl)) |i, sl| source_locations[i] = sl;
665 gop.value_ptr.source_locations = source_locations;
666
667 ws.coverage_condition.broadcast();
668}
669
670fn addEntryPoint(ws: *WebServer, coverage_id: u64, addr: u64) error{ AlreadyReported, OutOfMemory }!void {
671 ws.coverage_mutex.lock();
672 defer ws.coverage_mutex.unlock();
673
674 const coverage_map = ws.coverage_files.getPtr(coverage_id).?;
675 const header: *const abi.SeenPcsHeader = @ptrCast(coverage_map.mapped_memory[0..@sizeOf(abi.SeenPcsHeader)]);
676 const pcs = header.pcAddrs();
677 // Since this pcs list is unsorted, we must linear scan for the best index.
678 const index = i: {
679 var best: usize = 0;
680 for (pcs[1..], 1..) |elem_addr, i| {
681 if (elem_addr == addr) break :i i;
682 if (elem_addr > addr) continue;
683 if (elem_addr > pcs[best]) best = i;
684 }
685 break :i best;
686 };
687 if (index >= pcs.len) {
688 log.err("unable to find unit test entry address 0x{x} in source locations (range: 0x{x} to 0x{x})", .{
689 addr, pcs[0], pcs[pcs.len - 1],
690 });
691 return error.AlreadyReported;
692 }
693 if (false) {
694 const sl = coverage_map.source_locations[index];
695 const file_name = coverage_map.coverage.stringAt(coverage_map.coverage.fileAt(sl.file).basename);
696 log.debug("server found entry point for 0x{x} at {s}:{d}:{d} - index {d} between {x} and {x}", .{
697 addr, file_name, sl.line, sl.column, index, pcs[index - 1], pcs[index + 1],
698 });
699 }
700 const gpa = ws.gpa;
701 try coverage_map.entry_points.append(gpa, @intCast(index));
702}
703
704fn makeIov(s: []const u8) std.posix.iovec_const {
705 return .{
706 .base = s.ptr,
707 .len = s.len,
708 };
709}
lib/std/Build/Fuzz/abi.zig deleted-112
......@@ -1,112 +0,0 @@
1//! This file is shared among Zig code running in wildly different contexts:
2//! libfuzzer, compiled alongside unit tests, the build runner, running on the
3//! host computer, and the fuzzing web interface webassembly code running in
4//! the browser. All of these components interface to some degree via an ABI.
5
6/// libfuzzer uses this and its usize is the one that counts. To match the ABI,
7/// make the ints be the size of the target used with libfuzzer.
8///
9/// Trailing:
10/// * 1 bit per pc_addr, usize elements
11/// * pc_addr: usize for each pcs_len
12pub const SeenPcsHeader = extern struct {
13 n_runs: usize,
14 unique_runs: usize,
15 pcs_len: usize,
16
17 /// Used for comptime assertions. Provides a mechanism for strategically
18 /// causing compile errors.
19 pub const trailing = .{
20 .pc_bits_usize,
21 .pc_addr,
22 };
23
24 pub fn headerEnd(header: *const SeenPcsHeader) []const usize {
25 const ptr: [*]align(@alignOf(usize)) const u8 = @ptrCast(header);
26 const header_end_ptr: [*]const usize = @ptrCast(ptr + @sizeOf(SeenPcsHeader));
27 const pcs_len = header.pcs_len;
28 return header_end_ptr[0 .. pcs_len + seenElemsLen(pcs_len)];
29 }
30
31 pub fn seenBits(header: *const SeenPcsHeader) []const usize {
32 return header.headerEnd()[0..seenElemsLen(header.pcs_len)];
33 }
34
35 pub fn seenElemsLen(pcs_len: usize) usize {
36 return (pcs_len + @bitSizeOf(usize) - 1) / @bitSizeOf(usize);
37 }
38
39 pub fn pcAddrs(header: *const SeenPcsHeader) []const usize {
40 const pcs_len = header.pcs_len;
41 return header.headerEnd()[seenElemsLen(pcs_len)..][0..pcs_len];
42 }
43};
44
45pub const ToClientTag = enum(u8) {
46 current_time,
47 source_index,
48 coverage_update,
49 entry_points,
50 _,
51};
52
53pub const CurrentTime = extern struct {
54 tag: ToClientTag = .current_time,
55 /// Number of nanoseconds that all other timestamps are in reference to.
56 base: i64 align(1),
57};
58
59/// Sent to the fuzzer web client on first connection to the websocket URL.
60///
61/// Trailing:
62/// * std.debug.Coverage.String for each directories_len
63/// * std.debug.Coverage.File for each files_len
64/// * std.debug.Coverage.SourceLocation for each source_locations_len
65/// * u8 for each string_bytes_len
66pub const SourceIndexHeader = extern struct {
67 flags: Flags,
68 directories_len: u32,
69 files_len: u32,
70 source_locations_len: u32,
71 string_bytes_len: u32,
72 /// When, according to the server, fuzzing started.
73 start_timestamp: i64 align(4),
74
75 pub const Flags = packed struct(u32) {
76 tag: ToClientTag = .source_index,
77 _: u24 = 0,
78 };
79};
80
81/// Sent to the fuzzer web client whenever the set of covered source locations
82/// changes.
83///
84/// Trailing:
85/// * one bit per source_locations_len, contained in u64 elements
86pub const CoverageUpdateHeader = extern struct {
87 flags: Flags = .{},
88 n_runs: u64,
89 unique_runs: u64,
90
91 pub const Flags = packed struct(u64) {
92 tag: ToClientTag = .coverage_update,
93 _: u56 = 0,
94 };
95
96 pub const trailing = .{
97 .pc_bits_usize,
98 };
99};
100
101/// Sent to the fuzzer web client when the set of entry points is updated.
102///
103/// Trailing:
104/// * one u32 index of source_locations per locs_len
105pub const EntryPointHeader = extern struct {
106 flags: Flags,
107
108 pub const Flags = packed struct(u32) {
109 tag: ToClientTag = .entry_points,
110 locs_len: u24,
111 };
112};
lib/std/Build/Step.zig+55-17
......@@ -72,6 +72,14 @@ pub const MakeOptions = struct {
7272 progress_node: std.Progress.Node,
7373 thread_pool: *std.Thread.Pool,
7474 watch: bool,
75 web_server: switch (builtin.target.cpu.arch) {
76 else => ?*Build.WebServer,
77 // WASM code references `Build.abi` which happens to incidentally reference this type, but
78 // it currently breaks because `std.net.Address` doesn't work there. Work around for now.
79 .wasm32 => void,
80 },
81 /// Not to be confused with `Build.allocator`, which is an alias of `Build.graph.arena`.
82 gpa: Allocator,
7583};
7684
7785pub const MakeFn = *const fn (step: *Step, options: MakeOptions) anyerror!void;
......@@ -229,7 +237,17 @@ pub fn init(options: StepOptions) Step {
229237pub fn make(s: *Step, options: MakeOptions) error{ MakeFailed, MakeSkipped }!void {
230238 const arena = s.owner.allocator;
231239
232 s.makeFn(s, options) catch |err| switch (err) {
240 var timer: ?std.time.Timer = t: {
241 if (!s.owner.graph.time_report) break :t null;
242 if (s.id == .compile) break :t null;
243 break :t std.time.Timer.start() catch @panic("--time-report not supported on this host");
244 };
245 const make_result = s.makeFn(s, options);
246 if (timer) |*t| {
247 options.web_server.?.updateTimeReportGeneric(s, t.read());
248 }
249
250 make_result catch |err| switch (err) {
233251 error.MakeFailed => return error.MakeFailed,
234252 error.MakeSkipped => return error.MakeSkipped,
235253 else => {
......@@ -372,18 +390,20 @@ pub fn evalZigProcess(
372390 argv: []const []const u8,
373391 prog_node: std.Progress.Node,
374392 watch: bool,
393 web_server: ?*Build.WebServer,
394 gpa: Allocator,
375395) !?Path {
376396 if (s.getZigProcess()) |zp| update: {
377397 assert(watch);
378398 if (std.Progress.have_ipc) if (zp.progress_ipc_fd) |fd| prog_node.setIpcFd(fd);
379 const result = zigProcessUpdate(s, zp, watch) catch |err| switch (err) {
399 const result = zigProcessUpdate(s, zp, watch, web_server, gpa) catch |err| switch (err) {
380400 error.BrokenPipe => {
381401 // Process restart required.
382402 const term = zp.child.wait() catch |e| {
383403 return s.fail("unable to wait for {s}: {s}", .{ argv[0], @errorName(e) });
384404 };
385405 _ = term;
386 s.clearZigProcess();
406 s.clearZigProcess(gpa);
387407 break :update;
388408 },
389409 else => |e| return e,
......@@ -398,7 +418,7 @@ pub fn evalZigProcess(
398418 return s.fail("unable to wait for {s}: {s}", .{ argv[0], @errorName(e) });
399419 };
400420 s.result_peak_rss = zp.child.resource_usage_statistics.getMaxRss() orelse 0;
401 s.clearZigProcess();
421 s.clearZigProcess(gpa);
402422 try handleChildProcessTerm(s, term, null, argv);
403423 return error.MakeFailed;
404424 }
......@@ -408,7 +428,6 @@ pub fn evalZigProcess(
408428 assert(argv.len != 0);
409429 const b = s.owner;
410430 const arena = b.allocator;
411 const gpa = arena;
412431
413432 try handleChildProcUnsupported(s, null, argv);
414433 try handleVerbose(s.owner, null, argv);
......@@ -435,9 +454,12 @@ pub fn evalZigProcess(
435454 .progress_ipc_fd = if (std.Progress.have_ipc) child.progress_node.getIpcFd() else {},
436455 };
437456 if (watch) s.setZigProcess(zp);
438 defer if (!watch) zp.poller.deinit();
457 defer if (!watch) {
458 zp.poller.deinit();
459 gpa.destroy(zp);
460 };
439461
440 const result = try zigProcessUpdate(s, zp, watch);
462 const result = try zigProcessUpdate(s, zp, watch, web_server, gpa);
441463
442464 if (!watch) {
443465 // Send EOF to stdin.
......@@ -499,7 +521,7 @@ pub fn installDir(s: *Step, dest_path: []const u8) !std.fs.Dir.MakePathStatus {
499521 };
500522}
501523
502fn zigProcessUpdate(s: *Step, zp: *ZigProcess, watch: bool) !?Path {
524fn zigProcessUpdate(s: *Step, zp: *ZigProcess, watch: bool, web_server: ?*Build.WebServer, gpa: Allocator) !?Path {
503525 const b = s.owner;
504526 const arena = b.allocator;
505527
......@@ -537,12 +559,14 @@ fn zigProcessUpdate(s: *Step, zp: *ZigProcess, watch: bool) !?Path {
537559 body[@sizeOf(EbHdr) + extra_bytes.len ..][0..eb_hdr.string_bytes_len];
538560 // TODO: use @ptrCast when the compiler supports it
539561 const unaligned_extra = std.mem.bytesAsSlice(u32, extra_bytes);
540 const extra_array = try arena.alloc(u32, unaligned_extra.len);
541 @memcpy(extra_array, unaligned_extra);
542 s.result_error_bundle = .{
543 .string_bytes = try arena.dupe(u8, string_bytes),
544 .extra = extra_array,
545 };
562 {
563 s.result_error_bundle = .{ .string_bytes = &.{}, .extra = &.{} };
564 errdefer s.result_error_bundle.deinit(gpa);
565 s.result_error_bundle.string_bytes = try gpa.dupe(u8, string_bytes);
566 const extra = try gpa.alloc(u32, unaligned_extra.len);
567 @memcpy(extra, unaligned_extra);
568 s.result_error_bundle.extra = extra;
569 }
546570 // This message indicates the end of the update.
547571 if (watch) break :poll;
548572 },
......@@ -602,6 +626,20 @@ fn zigProcessUpdate(s: *Step, zp: *ZigProcess, watch: bool) !?Path {
602626 }
603627 }
604628 },
629 .time_report => if (web_server) |ws| {
630 const TimeReport = std.zig.Server.Message.TimeReport;
631 const tr: *align(1) const TimeReport = @ptrCast(body[0..@sizeOf(TimeReport)]);
632 ws.updateTimeReportCompile(.{
633 .compile = s.cast(Step.Compile).?,
634 .use_llvm = tr.flags.use_llvm,
635 .stats = tr.stats,
636 .ns_total = timer.read(),
637 .llvm_pass_timings_len = tr.llvm_pass_timings_len,
638 .files_len = tr.files_len,
639 .decls_len = tr.decls_len,
640 .trailing = body[@sizeOf(TimeReport)..],
641 });
642 },
605643 else => {}, // ignore other messages
606644 }
607645 }
......@@ -630,8 +668,7 @@ fn setZigProcess(s: *Step, zp: *ZigProcess) void {
630668 }
631669}
632670
633fn clearZigProcess(s: *Step) void {
634 const gpa = s.owner.allocator;
671fn clearZigProcess(s: *Step, gpa: Allocator) void {
635672 switch (s.id) {
636673 .compile => {
637674 const compile = s.cast(Compile).?;
......@@ -947,7 +984,8 @@ fn addWatchInputFromPath(step: *Step, path: Build.Cache.Path, basename: []const
947984 try gop.value_ptr.append(gpa, basename);
948985}
949986
950fn reset(step: *Step, gpa: Allocator) void {
987/// Implementation detail of file watching and forced rebuilds. Prepares the step for being re-evaluated.
988pub fn reset(step: *Step, gpa: Allocator) void {
951989 assert(step.state == .precheck_done);
952990
953991 step.result_error_msgs.clearRetainingCapacity();
lib/std/Build/Step/Compile.zig+5-4
......@@ -1491,6 +1491,7 @@ fn getZigArgs(compile: *Compile, fuzz: bool) ![][]const u8 {
14911491 if (b.verbose_link or compile.verbose_link) try zig_args.append("--verbose-link");
14921492 if (b.verbose_cc or compile.verbose_cc) try zig_args.append("--verbose-cc");
14931493 if (b.verbose_llvm_cpu_features) try zig_args.append("--verbose-llvm-cpu-features");
1494 if (b.graph.time_report) try zig_args.append("--time-report");
14941495
14951496 if (compile.generated_asm != null) try zig_args.append("-femit-asm");
14961497 if (compile.generated_bin == null) try zig_args.append("-fno-emit-bin");
......@@ -1851,6 +1852,8 @@ fn make(step: *Step, options: Step.MakeOptions) !void {
18511852 zig_args,
18521853 options.progress_node,
18531854 (b.graph.incremental == true) and options.watch,
1855 options.web_server,
1856 options.gpa,
18541857 ) catch |err| switch (err) {
18551858 error.NeedCompileErrorCheck => {
18561859 assert(compile.expect_errors != null);
......@@ -1905,9 +1908,7 @@ fn outputPath(c: *Compile, out_dir: std.Build.Cache.Path, ea: std.zig.EmitArtifa
19051908 return out_dir.joinString(arena, name) catch @panic("OOM");
19061909}
19071910
1908pub fn rebuildInFuzzMode(c: *Compile, progress_node: std.Progress.Node) !Path {
1909 const gpa = c.step.owner.allocator;
1910
1911pub fn rebuildInFuzzMode(c: *Compile, gpa: Allocator, progress_node: std.Progress.Node) !Path {
19111912 c.step.result_error_msgs.clearRetainingCapacity();
19121913 c.step.result_stderr = "";
19131914
......@@ -1915,7 +1916,7 @@ pub fn rebuildInFuzzMode(c: *Compile, progress_node: std.Progress.Node) !Path {
19151916 c.step.result_error_bundle = std.zig.ErrorBundle.empty;
19161917
19171918 const zig_args = try getZigArgs(c, true);
1918 const maybe_output_bin_path = try c.step.evalZigProcess(zig_args, progress_node, false);
1919 const maybe_output_bin_path = try c.step.evalZigProcess(zig_args, progress_node, false, null, gpa);
19191920 return maybe_output_bin_path.?;
19201921}
19211922
lib/std/Build/Step/ObjCopy.zig+1-1
......@@ -236,7 +236,7 @@ fn make(step: *Step, options: Step.MakeOptions) !void {
236236 try argv.appendSlice(&.{ full_src_path, full_dest_path });
237237
238238 try argv.append("--listen=-");
239 _ = try step.evalZigProcess(argv.items, prog_node, false);
239 _ = try step.evalZigProcess(argv.items, prog_node, false, options.web_server, options.gpa);
240240
241241 objcopy.output_file.path = full_dest_path;
242242 if (objcopy.output_file_debug) |*file| file.path = full_dest_path_debug;
lib/std/Build/Step/Options.zig+1
......@@ -549,6 +549,7 @@ test Options {
549549 .result = try std.zig.system.resolveTargetQuery(.{}),
550550 },
551551 .zig_lib_directory = std.Build.Cache.Directory.cwd(),
552 .time_report = false,
552553 };
553554
554555 var builder = try std.Build.create(
lib/std/Build/Step/Run.zig+13-13
......@@ -944,7 +944,7 @@ fn make(step: *Step, options: Step.MakeOptions) !void {
944944
945945pub fn rerunInFuzzMode(
946946 run: *Run,
947 web_server: *std.Build.Fuzz.WebServer,
947 fuzz: *std.Build.Fuzz,
948948 unit_test_index: u32,
949949 prog_node: std.Progress.Node,
950950) !void {
......@@ -984,7 +984,7 @@ pub fn rerunInFuzzMode(
984984 const tmp_dir_path = "tmp" ++ fs.path.sep_str ++ std.fmt.hex(rand_int);
985985 try runCommand(run, argv_list.items, has_side_effects, tmp_dir_path, prog_node, .{
986986 .unit_test_index = unit_test_index,
987 .web_server = web_server,
987 .fuzz = fuzz,
988988 });
989989}
990990
......@@ -1054,7 +1054,7 @@ fn termMatches(expected: ?std.process.Child.Term, actual: std.process.Child.Term
10541054}
10551055
10561056const FuzzContext = struct {
1057 web_server: *std.Build.Fuzz.WebServer,
1057 fuzz: *std.Build.Fuzz,
10581058 unit_test_index: u32,
10591059};
10601060
......@@ -1638,31 +1638,31 @@ fn evalZigTest(
16381638 };
16391639 },
16401640 .coverage_id => {
1641 const web_server = fuzz_context.?.web_server;
1641 const fuzz = fuzz_context.?.fuzz;
16421642 const msg_ptr: *align(1) const u64 = @ptrCast(body);
16431643 coverage_id = msg_ptr.*;
16441644 {
1645 web_server.mutex.lock();
1646 defer web_server.mutex.unlock();
1647 try web_server.msg_queue.append(web_server.gpa, .{ .coverage = .{
1645 fuzz.queue_mutex.lock();
1646 defer fuzz.queue_mutex.unlock();
1647 try fuzz.msg_queue.append(fuzz.ws.gpa, .{ .coverage = .{
16481648 .id = coverage_id.?,
16491649 .run = run,
16501650 } });
1651 web_server.condition.signal();
1651 fuzz.queue_cond.signal();
16521652 }
16531653 },
16541654 .fuzz_start_addr => {
1655 const web_server = fuzz_context.?.web_server;
1655 const fuzz = fuzz_context.?.fuzz;
16561656 const msg_ptr: *align(1) const u64 = @ptrCast(body);
16571657 const addr = msg_ptr.*;
16581658 {
1659 web_server.mutex.lock();
1660 defer web_server.mutex.unlock();
1661 try web_server.msg_queue.append(web_server.gpa, .{ .entry_point = .{
1659 fuzz.queue_mutex.lock();
1660 defer fuzz.queue_mutex.unlock();
1661 try fuzz.msg_queue.append(fuzz.ws.gpa, .{ .entry_point = .{
16621662 .addr = addr,
16631663 .coverage_id = coverage_id.?,
16641664 } });
1665 web_server.condition.signal();
1665 fuzz.queue_cond.signal();
16661666 }
16671667 },
16681668 else => {}, // ignore other messages
lib/std/Build/Step/TranslateC.zig+1-1
......@@ -187,7 +187,7 @@ fn make(step: *Step, options: Step.MakeOptions) !void {
187187 const c_source_path = translate_c.source.getPath2(b, step);
188188 try argv_list.append(c_source_path);
189189
190 const output_dir = try step.evalZigProcess(argv_list.items, prog_node, false);
190 const output_dir = try step.evalZigProcess(argv_list.items, prog_node, false, options.web_server, options.gpa);
191191
192192 const basename = std.fs.path.stem(std.fs.path.basename(c_source_path));
193193 translate_c.out_basename = b.fmt("{s}.zig", .{basename});
lib/std/Build/WebServer.zig created+823
......@@ -0,0 +1,823 @@
1gpa: Allocator,
2thread_pool: *std.Thread.Pool,
3graph: *const Build.Graph,
4all_steps: []const *Build.Step,
5listen_address: std.net.Address,
6ttyconf: std.io.tty.Config,
7root_prog_node: std.Progress.Node,
8watch: bool,
9
10tcp_server: ?std.net.Server,
11serve_thread: ?std.Thread,
12
13base_timestamp: i128,
14/// The "step name" data which trails `abi.Hello`, for the steps in `all_steps`.
15step_names_trailing: []u8,
16
17/// The bit-packed "step status" data. Values are `abi.StepUpdate.Status`. LSBs are earlier steps.
18/// Accessed atomically.
19step_status_bits: []u8,
20
21fuzz: ?Fuzz,
22time_report_mutex: std.Thread.Mutex,
23time_report_msgs: [][]u8,
24time_report_update_times: []i64,
25
26build_status: std.atomic.Value(abi.BuildStatus),
27/// When an event occurs which means WebSocket clients should be sent updates, call `notifyUpdate`
28/// to increment this value. Each client thread waits for this increment with `std.Thread.Futex`, so
29/// `notifyUpdate` will wake those threads. Updates are sent on a short interval regardless, so it
30/// is recommended to only use `notifyUpdate` for changes which the user should see immediately. For
31/// instance, we do not call `notifyUpdate` when the number of "unique runs" in the fuzzer changes,
32/// because this value changes quickly so this would result in constantly spamming all clients with
33/// an unreasonable number of packets.
34update_id: std.atomic.Value(u32),
35
36runner_request_mutex: std.Thread.Mutex,
37runner_request_ready_cond: std.Thread.Condition,
38runner_request_empty_cond: std.Thread.Condition,
39runner_request: ?RunnerRequest,
40
41/// If a client is not explicitly notified of changes with `notifyUpdate`, it will be sent updates
42/// on a fixed interval of this many milliseconds.
43const default_update_interval_ms = 500;
44
45/// Thread-safe. Triggers updates to be sent to connected WebSocket clients; see `update_id`.
46pub fn notifyUpdate(ws: *WebServer) void {
47 _ = ws.update_id.rmw(.Add, 1, .release);
48 std.Thread.Futex.wake(&ws.update_id, 16);
49}
50
51pub const Options = struct {
52 gpa: Allocator,
53 thread_pool: *std.Thread.Pool,
54 graph: *const std.Build.Graph,
55 all_steps: []const *Build.Step,
56 ttyconf: std.io.tty.Config,
57 root_prog_node: std.Progress.Node,
58 watch: bool,
59 listen_address: std.net.Address,
60};
61pub fn init(opts: Options) WebServer {
62 if (builtin.single_threaded) {
63 // The upcoming `std.Io` interface should allow us to use `Io.async` and `Io.concurrent`
64 // instead of threads, so that the web server can function in single-threaded builds.
65 std.process.fatal("--webui not yet implemented for single-threaded builds", .{});
66 }
67
68 if (builtin.os.tag == .windows) {
69 // At the time of writing, there are two bugs in the standard library which break this feature on Windows:
70 // * Reading from a socket on one thread while writing to it on another seems to deadlock.
71 // * Vectored writes to sockets currently trigger an infinite loop when a buffer has length 0.
72 //
73 // Both of these bugs are expected to be solved by changes which are currently in the unmerged
74 // 'wrangle-writer-buffering' branch. Until that makes it in, this must remain disabled.
75 std.process.fatal("--webui is currently disabled on Windows due to bugs", .{});
76 }
77
78 const all_steps = opts.all_steps;
79
80 const step_names_trailing = opts.gpa.alloc(u8, len: {
81 var name_bytes: usize = 0;
82 for (all_steps) |step| name_bytes += step.name.len;
83 break :len name_bytes + all_steps.len * 4;
84 }) catch @panic("out of memory");
85 {
86 const step_name_lens: []align(1) u32 = @ptrCast(step_names_trailing[0 .. all_steps.len * 4]);
87 var idx: usize = all_steps.len * 4;
88 for (all_steps, step_name_lens) |step, *name_len| {
89 name_len.* = @intCast(step.name.len);
90 @memcpy(step_names_trailing[idx..][0..step.name.len], step.name);
91 idx += step.name.len;
92 }
93 assert(idx == step_names_trailing.len);
94 }
95
96 const step_status_bits = opts.gpa.alloc(
97 u8,
98 std.math.divCeil(usize, all_steps.len, 4) catch unreachable,
99 ) catch @panic("out of memory");
100 @memset(step_status_bits, 0);
101
102 const time_reports_len: usize = if (opts.graph.time_report) all_steps.len else 0;
103 const time_report_msgs = opts.gpa.alloc([]u8, time_reports_len) catch @panic("out of memory");
104 const time_report_update_times = opts.gpa.alloc(i64, time_reports_len) catch @panic("out of memory");
105 @memset(time_report_msgs, &.{});
106 @memset(time_report_update_times, std.math.minInt(i64));
107
108 return .{
109 .gpa = opts.gpa,
110 .thread_pool = opts.thread_pool,
111 .graph = opts.graph,
112 .all_steps = all_steps,
113 .listen_address = opts.listen_address,
114 .ttyconf = opts.ttyconf,
115 .root_prog_node = opts.root_prog_node,
116 .watch = opts.watch,
117
118 .tcp_server = null,
119 .serve_thread = null,
120
121 .base_timestamp = std.time.nanoTimestamp(),
122 .step_names_trailing = step_names_trailing,
123
124 .step_status_bits = step_status_bits,
125
126 .fuzz = null,
127 .time_report_mutex = .{},
128 .time_report_msgs = time_report_msgs,
129 .time_report_update_times = time_report_update_times,
130
131 .build_status = .init(.idle),
132 .update_id = .init(0),
133
134 .runner_request_mutex = .{},
135 .runner_request_ready_cond = .{},
136 .runner_request_empty_cond = .{},
137 .runner_request = null,
138 };
139}
140pub fn deinit(ws: *WebServer) void {
141 const gpa = ws.gpa;
142
143 gpa.free(ws.step_names_trailing);
144 gpa.free(ws.step_status_bits);
145
146 if (ws.fuzz) |*f| f.deinit();
147 for (ws.time_report_msgs) |msg| gpa.free(msg);
148 gpa.free(ws.time_report_msgs);
149 gpa.free(ws.time_report_update_times);
150
151 if (ws.serve_thread) |t| {
152 if (ws.tcp_server) |*s| s.stream.close();
153 t.join();
154 }
155 if (ws.tcp_server) |*s| s.deinit();
156
157 gpa.free(ws.step_names_trailing);
158}
159pub fn start(ws: *WebServer) error{AlreadyReported}!void {
160 assert(ws.tcp_server == null);
161 assert(ws.serve_thread == null);
162
163 ws.tcp_server = ws.listen_address.listen(.{ .reuse_address = true }) catch |err| {
164 log.err("failed to listen to port {d}: {s}", .{ ws.listen_address.getPort(), @errorName(err) });
165 return error.AlreadyReported;
166 };
167 ws.serve_thread = std.Thread.spawn(.{}, serve, .{ws}) catch |err| {
168 log.err("unable to spawn web server thread: {s}", .{@errorName(err)});
169 ws.tcp_server.?.deinit();
170 ws.tcp_server = null;
171 return error.AlreadyReported;
172 };
173
174 log.info("web interface listening at http://{f}/", .{ws.tcp_server.?.listen_address});
175 if (ws.listen_address.getPort() == 0) {
176 log.info("hint: pass '--webui={f}' to use the same port next time", .{ws.tcp_server.?.listen_address});
177 }
178}
179fn serve(ws: *WebServer) void {
180 while (true) {
181 const connection = ws.tcp_server.?.accept() catch |err| {
182 log.err("failed to accept connection: {s}", .{@errorName(err)});
183 return;
184 };
185 _ = std.Thread.spawn(.{}, accept, .{ ws, connection }) catch |err| {
186 log.err("unable to spawn connection thread: {s}", .{@errorName(err)});
187 connection.stream.close();
188 continue;
189 };
190 }
191}
192
193pub fn startBuild(ws: *WebServer) void {
194 if (ws.fuzz) |*fuzz| {
195 fuzz.deinit();
196 ws.fuzz = null;
197 }
198 for (ws.step_status_bits) |*bits| @atomicStore(u8, bits, 0, .monotonic);
199 ws.build_status.store(.running, .monotonic);
200 ws.notifyUpdate();
201}
202
203pub fn updateStepStatus(ws: *WebServer, step: *Build.Step, new_status: abi.StepUpdate.Status) void {
204 const step_idx: u32 = for (ws.all_steps, 0..) |s, i| {
205 if (s == step) break @intCast(i);
206 } else unreachable;
207 const ptr = &ws.step_status_bits[step_idx / 4];
208 const bit_offset: u3 = @intCast((step_idx % 4) * 2);
209 const old_bits: u2 = @truncate(@atomicLoad(u8, ptr, .monotonic) >> bit_offset);
210 const mask = @as(u8, @intFromEnum(new_status) ^ old_bits) << bit_offset;
211 _ = @atomicRmw(u8, ptr, .Xor, mask, .monotonic);
212 ws.notifyUpdate();
213}
214
215pub fn finishBuild(ws: *WebServer, opts: struct {
216 fuzz: bool,
217}) void {
218 if (opts.fuzz) {
219 switch (builtin.os.tag) {
220 // Current implementation depends on two things that need to be ported to Windows:
221 // * Memory-mapping to share data between the fuzzer and build runner.
222 // * COFF/PE support added to `std.debug.Info` (it needs a batching API for resolving
223 // many addresses to source locations).
224 .windows => std.process.fatal("--fuzz not yet implemented for {s}", .{@tagName(builtin.os.tag)}),
225 else => {},
226 }
227 if (@bitSizeOf(usize) != 64) {
228 // Current implementation depends on posix.mmap()'s second parameter, `length: usize`,
229 // being compatible with `std.fs.getEndPos() u64`'s return value. This is not the case
230 // on 32-bit platforms.
231 // Affects or affected by issues #5185, #22523, and #22464.
232 std.process.fatal("--fuzz not yet implemented on {d}-bit platforms", .{@bitSizeOf(usize)});
233 }
234 assert(ws.fuzz == null);
235
236 ws.build_status.store(.fuzz_init, .monotonic);
237 ws.notifyUpdate();
238
239 ws.fuzz = Fuzz.init(ws) catch |err| std.process.fatal("failed to start fuzzer: {s}", .{@errorName(err)});
240 ws.fuzz.?.start();
241 }
242
243 ws.build_status.store(if (ws.watch) .watching else .idle, .monotonic);
244 ws.notifyUpdate();
245}
246
247pub fn now(s: *const WebServer) i64 {
248 return @intCast(std.time.nanoTimestamp() - s.base_timestamp);
249}
250
251fn accept(ws: *WebServer, connection: std.net.Server.Connection) void {
252 defer connection.stream.close();
253
254 var read_buf: [0x4000]u8 = undefined;
255 var server: std.http.Server = .init(connection, &read_buf);
256
257 while (true) {
258 var request = server.receiveHead() catch |err| switch (err) {
259 error.HttpConnectionClosing => return,
260 else => {
261 log.err("failed to receive http request: {s}", .{@errorName(err)});
262 return;
263 },
264 };
265 var ws_send_buf: [0x4000]u8 = undefined;
266 var ws_recv_buf: [0x4000]u8 align(4) = undefined;
267 if (std.http.WebSocket.init(&request, &ws_send_buf, &ws_recv_buf) catch |err| {
268 log.err("failed to initialize websocket connection: {s}", .{@errorName(err)});
269 return;
270 }) |ws_init| {
271 var web_socket = ws_init;
272 ws.serveWebSocket(&web_socket) catch |err| {
273 log.err("failed to serve websocket: {s}", .{@errorName(err)});
274 return;
275 };
276 comptime unreachable;
277 } else {
278 ws.serveRequest(&request) catch |err| switch (err) {
279 error.AlreadyReported => return,
280 else => {
281 log.err("failed to serve '{s}': {s}", .{ request.head.target, @errorName(err) });
282 return;
283 },
284 };
285 }
286 }
287}
288
289fn makeIov(s: []const u8) std.posix.iovec_const {
290 return .{
291 .base = s.ptr,
292 .len = s.len,
293 };
294}
295fn serveWebSocket(ws: *WebServer, sock: *std.http.WebSocket) !noreturn {
296 var prev_build_status = ws.build_status.load(.monotonic);
297
298 const prev_step_status_bits = try ws.gpa.alloc(u8, ws.step_status_bits.len);
299 defer ws.gpa.free(prev_step_status_bits);
300 for (prev_step_status_bits, ws.step_status_bits) |*copy, *shared| {
301 copy.* = @atomicLoad(u8, shared, .monotonic);
302 }
303
304 _ = try std.Thread.spawn(.{}, recvWebSocketMessages, .{ ws, sock });
305
306 {
307 const hello_header: abi.Hello = .{
308 .status = prev_build_status,
309 .flags = .{
310 .time_report = ws.graph.time_report,
311 },
312 .timestamp = ws.now(),
313 .steps_len = @intCast(ws.all_steps.len),
314 };
315 try sock.writeMessagev(&.{
316 makeIov(@ptrCast(&hello_header)),
317 makeIov(ws.step_names_trailing),
318 makeIov(prev_step_status_bits),
319 }, .binary);
320 }
321
322 var prev_fuzz: Fuzz.Previous = .init;
323 var prev_time: i64 = std.math.minInt(i64);
324 while (true) {
325 const start_time = ws.now();
326 const start_update_id = ws.update_id.load(.acquire);
327
328 if (ws.fuzz) |*fuzz| {
329 try fuzz.sendUpdate(sock, &prev_fuzz);
330 }
331
332 {
333 ws.time_report_mutex.lock();
334 defer ws.time_report_mutex.unlock();
335 for (ws.time_report_msgs, ws.time_report_update_times) |msg, update_time| {
336 if (update_time <= prev_time) continue;
337 // We want to send `msg`, but shouldn't block `ws.time_report_mutex` while we do, so
338 // that we don't hold up the build system on the client accepting this packet.
339 const owned_msg = try ws.gpa.dupe(u8, msg);
340 defer ws.gpa.free(owned_msg);
341 // Temporarily unlock, then re-lock after the message is sent.
342 ws.time_report_mutex.unlock();
343 defer ws.time_report_mutex.lock();
344 try sock.writeMessage(msg, .binary);
345 }
346 }
347
348 {
349 const build_status = ws.build_status.load(.monotonic);
350 if (build_status != prev_build_status) {
351 prev_build_status = build_status;
352 const msg: abi.StatusUpdate = .{ .new = build_status };
353 try sock.writeMessage(@ptrCast(&msg), .binary);
354 }
355 }
356
357 for (prev_step_status_bits, ws.step_status_bits, 0..) |*prev_byte, *shared, byte_idx| {
358 const cur_byte = @atomicLoad(u8, shared, .monotonic);
359 if (prev_byte.* == cur_byte) continue;
360 const cur: [4]abi.StepUpdate.Status = .{
361 @enumFromInt(@as(u2, @truncate(cur_byte >> 0))),
362 @enumFromInt(@as(u2, @truncate(cur_byte >> 2))),
363 @enumFromInt(@as(u2, @truncate(cur_byte >> 4))),
364 @enumFromInt(@as(u2, @truncate(cur_byte >> 6))),
365 };
366 const prev: [4]abi.StepUpdate.Status = .{
367 @enumFromInt(@as(u2, @truncate(prev_byte.* >> 0))),
368 @enumFromInt(@as(u2, @truncate(prev_byte.* >> 2))),
369 @enumFromInt(@as(u2, @truncate(prev_byte.* >> 4))),
370 @enumFromInt(@as(u2, @truncate(prev_byte.* >> 6))),
371 };
372 for (cur, prev, byte_idx * 4..) |cur_status, prev_status, step_idx| {
373 const msg: abi.StepUpdate = .{ .step_idx = @intCast(step_idx), .bits = .{ .status = cur_status } };
374 if (cur_status != prev_status) try sock.writeMessage(@ptrCast(&msg), .binary);
375 }
376 prev_byte.* = cur_byte;
377 }
378
379 prev_time = start_time;
380 std.Thread.Futex.timedWait(&ws.update_id, start_update_id, std.time.ns_per_ms * default_update_interval_ms) catch {};
381 }
382}
383fn recvWebSocketMessages(ws: *WebServer, sock: *std.http.WebSocket) void {
384 while (true) {
385 const msg = sock.readSmallMessage() catch return;
386 if (msg.opcode != .binary) continue;
387 if (msg.data.len == 0) continue;
388 const tag: abi.ToServerTag = @enumFromInt(msg.data[0]);
389 switch (tag) {
390 _ => continue,
391 .rebuild => while (true) {
392 ws.runner_request_mutex.lock();
393 defer ws.runner_request_mutex.unlock();
394 if (ws.runner_request == null) {
395 ws.runner_request = .rebuild;
396 ws.runner_request_ready_cond.signal();
397 break;
398 }
399 ws.runner_request_empty_cond.wait(&ws.runner_request_mutex);
400 },
401 }
402 }
403}
404
405fn serveRequest(ws: *WebServer, req: *std.http.Server.Request) !void {
406 // Strip an optional leading '/debug' component from the request.
407 const target: []const u8, const debug: bool = target: {
408 if (mem.eql(u8, req.head.target, "/debug")) break :target .{ "/", true };
409 if (mem.eql(u8, req.head.target, "/debug/")) break :target .{ "/", true };
410 if (mem.startsWith(u8, req.head.target, "/debug/")) break :target .{ req.head.target["/debug".len..], true };
411 break :target .{ req.head.target, false };
412 };
413
414 if (mem.eql(u8, target, "/")) return serveLibFile(ws, req, "build-web/index.html", "text/html");
415 if (mem.eql(u8, target, "/main.js")) return serveLibFile(ws, req, "build-web/main.js", "application/javascript");
416 if (mem.eql(u8, target, "/style.css")) return serveLibFile(ws, req, "build-web/style.css", "text/css");
417 if (mem.eql(u8, target, "/time_report.css")) return serveLibFile(ws, req, "build-web/time_report.css", "text/css");
418 if (mem.eql(u8, target, "/main.wasm")) return serveClientWasm(ws, req, if (debug) .Debug else .ReleaseFast);
419
420 if (ws.fuzz) |*fuzz| {
421 if (mem.eql(u8, target, "/sources.tar")) return fuzz.serveSourcesTar(req);
422 }
423
424 try req.respond("not found", .{
425 .status = .not_found,
426 .extra_headers = &.{
427 .{ .name = "Content-Type", .value = "text/plain" },
428 },
429 });
430}
431
432fn serveLibFile(
433 ws: *WebServer,
434 request: *std.http.Server.Request,
435 sub_path: []const u8,
436 content_type: []const u8,
437) !void {
438 return serveFile(ws, request, .{
439 .root_dir = ws.graph.zig_lib_directory,
440 .sub_path = sub_path,
441 }, content_type);
442}
443fn serveClientWasm(
444 ws: *WebServer,
445 req: *std.http.Server.Request,
446 optimize_mode: std.builtin.OptimizeMode,
447) !void {
448 var arena_state: std.heap.ArenaAllocator = .init(ws.gpa);
449 defer arena_state.deinit();
450 const arena = arena_state.allocator();
451
452 // We always rebuild the wasm on-the-fly, so that if it is edited the user can just refresh the page.
453 const bin_path = try buildClientWasm(ws, arena, optimize_mode);
454 return serveFile(ws, req, bin_path, "application/wasm");
455}
456
457pub fn serveFile(
458 ws: *WebServer,
459 request: *std.http.Server.Request,
460 path: Cache.Path,
461 content_type: []const u8,
462) !void {
463 const gpa = ws.gpa;
464 // The desired API is actually sendfile, which will require enhancing std.http.Server.
465 // We load the file with every request so that the user can make changes to the file
466 // and refresh the HTML page without restarting this server.
467 const file_contents = path.root_dir.handle.readFileAlloc(gpa, path.sub_path, 10 * 1024 * 1024) catch |err| {
468 log.err("failed to read '{f}': {s}", .{ path, @errorName(err) });
469 return error.AlreadyReported;
470 };
471 defer gpa.free(file_contents);
472 try request.respond(file_contents, .{
473 .extra_headers = &.{
474 .{ .name = "Content-Type", .value = content_type },
475 cache_control_header,
476 },
477 });
478}
479pub fn serveTarFile(
480 ws: *WebServer,
481 request: *std.http.Server.Request,
482 paths: []const Cache.Path,
483) !void {
484 const gpa = ws.gpa;
485
486 var send_buf: [0x4000]u8 = undefined;
487 var response = request.respondStreaming(.{
488 .send_buffer = &send_buf,
489 .respond_options = .{
490 .extra_headers = &.{
491 .{ .name = "Content-Type", .value = "application/x-tar" },
492 cache_control_header,
493 },
494 },
495 });
496
497 var cached_cwd_path: ?[]const u8 = null;
498 defer if (cached_cwd_path) |p| gpa.free(p);
499
500 var response_buf: [1024]u8 = undefined;
501 var adapter = response.writer().adaptToNewApi();
502 adapter.new_interface.buffer = &response_buf;
503 var archiver: std.tar.Writer = .{ .underlying_writer = &adapter.new_interface };
504
505 for (paths) |path| {
506 var file = path.root_dir.handle.openFile(path.sub_path, .{}) catch |err| {
507 log.err("failed to open '{f}': {s}", .{ path, @errorName(err) });
508 continue;
509 };
510 defer file.close();
511 const stat = try file.stat();
512 var read_buffer: [1024]u8 = undefined;
513 var file_reader: std.fs.File.Reader = .initSize(file, &read_buffer, stat.size);
514
515 // TODO: this logic is completely bogus -- obviously so, because `path.root_dir.path` can
516 // be cwd-relative. This is also related to why linkification doesn't work in the fuzzer UI:
517 // it turns out the WASM treats the first path component as the module name, typically
518 // resulting in modules named "" and "src". The compiler needs to tell the build system
519 // about the module graph so that the build system can correctly encode this information in
520 // the tar file.
521 archiver.prefix = path.root_dir.path orelse cwd: {
522 if (cached_cwd_path == null) cached_cwd_path = try std.process.getCwdAlloc(gpa);
523 break :cwd cached_cwd_path.?;
524 };
525 try archiver.writeFile(path.sub_path, &file_reader, stat.mtime);
526 }
527
528 // intentionally not calling `archiver.finishPedantically`
529 try adapter.new_interface.flush();
530 try response.end();
531}
532
533fn buildClientWasm(ws: *WebServer, arena: Allocator, optimize: std.builtin.OptimizeMode) !Cache.Path {
534 const root_name = "build-web";
535 const arch_os_abi = "wasm32-freestanding";
536 const cpu_features = "baseline+atomics+bulk_memory+multivalue+mutable_globals+nontrapping_fptoint+reference_types+sign_ext";
537
538 const gpa = ws.gpa;
539 const graph = ws.graph;
540
541 const main_src_path: Cache.Path = .{
542 .root_dir = graph.zig_lib_directory,
543 .sub_path = "build-web/main.zig",
544 };
545 const walk_src_path: Cache.Path = .{
546 .root_dir = graph.zig_lib_directory,
547 .sub_path = "docs/wasm/Walk.zig",
548 };
549 const html_render_src_path: Cache.Path = .{
550 .root_dir = graph.zig_lib_directory,
551 .sub_path = "docs/wasm/html_render.zig",
552 };
553
554 var argv: std.ArrayListUnmanaged([]const u8) = .empty;
555
556 try argv.appendSlice(arena, &.{
557 graph.zig_exe, "build-exe", //
558 "-fno-entry", //
559 "-O", @tagName(optimize), //
560 "-target", arch_os_abi, //
561 "-mcpu", cpu_features, //
562 "--cache-dir", graph.global_cache_root.path orelse ".", //
563 "--global-cache-dir", graph.global_cache_root.path orelse ".", //
564 "--zig-lib-dir", graph.zig_lib_directory.path orelse ".", //
565 "--name", root_name, //
566 "-rdynamic", //
567 "-fsingle-threaded", //
568 "--dep", "Walk", //
569 "--dep", "html_render", //
570 try std.fmt.allocPrint(arena, "-Mroot={f}", .{main_src_path}), //
571 try std.fmt.allocPrint(arena, "-MWalk={f}", .{walk_src_path}), //
572 "--dep", "Walk", //
573 try std.fmt.allocPrint(arena, "-Mhtml_render={f}", .{html_render_src_path}), //
574 "--listen=-",
575 });
576
577 var child: std.process.Child = .init(argv.items, gpa);
578 child.stdin_behavior = .Pipe;
579 child.stdout_behavior = .Pipe;
580 child.stderr_behavior = .Pipe;
581 try child.spawn();
582
583 var poller = std.io.poll(gpa, enum { stdout, stderr }, .{
584 .stdout = child.stdout.?,
585 .stderr = child.stderr.?,
586 });
587 defer poller.deinit();
588
589 try child.stdin.?.writeAll(@ptrCast(@as([]const std.zig.Client.Message.Header, &.{
590 .{ .tag = .update, .bytes_len = 0 },
591 .{ .tag = .exit, .bytes_len = 0 },
592 })));
593
594 const Header = std.zig.Server.Message.Header;
595 var result: ?Cache.Path = null;
596 var result_error_bundle = std.zig.ErrorBundle.empty;
597
598 const stdout = poller.reader(.stdout);
599
600 poll: while (true) {
601 while (stdout.buffered().len < @sizeOf(Header)) if (!(try poller.poll())) break :poll;
602 const header = stdout.takeStruct(Header, .little) catch unreachable;
603 while (stdout.buffered().len < header.bytes_len) if (!try poller.poll()) break :poll;
604 const body = stdout.take(header.bytes_len) catch unreachable;
605
606 switch (header.tag) {
607 .zig_version => {
608 if (!std.mem.eql(u8, builtin.zig_version_string, body)) {
609 return error.ZigProtocolVersionMismatch;
610 }
611 },
612 .error_bundle => {
613 const EbHdr = std.zig.Server.Message.ErrorBundle;
614 const eb_hdr = @as(*align(1) const EbHdr, @ptrCast(body));
615 const extra_bytes =
616 body[@sizeOf(EbHdr)..][0 .. @sizeOf(u32) * eb_hdr.extra_len];
617 const string_bytes =
618 body[@sizeOf(EbHdr) + extra_bytes.len ..][0..eb_hdr.string_bytes_len];
619 const unaligned_extra: []align(1) const u32 = @ptrCast(extra_bytes);
620 const extra_array = try arena.alloc(u32, unaligned_extra.len);
621 @memcpy(extra_array, unaligned_extra);
622 result_error_bundle = .{
623 .string_bytes = try arena.dupe(u8, string_bytes),
624 .extra = extra_array,
625 };
626 },
627 .emit_digest => {
628 const EmitDigest = std.zig.Server.Message.EmitDigest;
629 const ebp_hdr: *align(1) const EmitDigest = @ptrCast(body);
630 if (!ebp_hdr.flags.cache_hit) {
631 log.info("source changes detected; rebuilt wasm component", .{});
632 }
633 const digest = body[@sizeOf(EmitDigest)..][0..Cache.bin_digest_len];
634 result = .{
635 .root_dir = graph.global_cache_root,
636 .sub_path = try arena.dupe(u8, "o" ++ std.fs.path.sep_str ++ Cache.binToHex(digest.*)),
637 };
638 },
639 else => {}, // ignore other messages
640 }
641 }
642
643 const stderr_contents = try poller.toOwnedSlice(.stderr);
644 if (stderr_contents.len > 0) {
645 std.debug.print("{s}", .{stderr_contents});
646 }
647
648 // Send EOF to stdin.
649 child.stdin.?.close();
650 child.stdin = null;
651
652 switch (try child.wait()) {
653 .Exited => |code| {
654 if (code != 0) {
655 log.err(
656 "the following command exited with error code {d}:\n{s}",
657 .{ code, try Build.Step.allocPrintCmd(arena, null, argv.items) },
658 );
659 return error.WasmCompilationFailed;
660 }
661 },
662 .Signal, .Stopped, .Unknown => {
663 log.err(
664 "the following command terminated unexpectedly:\n{s}",
665 .{try Build.Step.allocPrintCmd(arena, null, argv.items)},
666 );
667 return error.WasmCompilationFailed;
668 },
669 }
670
671 if (result_error_bundle.errorMessageCount() > 0) {
672 const color = std.zig.Color.auto;
673 result_error_bundle.renderToStdErr(color.renderOptions());
674 log.err("the following command failed with {d} compilation errors:\n{s}", .{
675 result_error_bundle.errorMessageCount(),
676 try Build.Step.allocPrintCmd(arena, null, argv.items),
677 });
678 return error.WasmCompilationFailed;
679 }
680
681 const base_path = result orelse {
682 log.err("child process failed to report result\n{s}", .{
683 try Build.Step.allocPrintCmd(arena, null, argv.items),
684 });
685 return error.WasmCompilationFailed;
686 };
687 const bin_name = try std.zig.binNameAlloc(arena, .{
688 .root_name = root_name,
689 .target = &(std.zig.system.resolveTargetQuery(std.Build.parseTargetQuery(.{
690 .arch_os_abi = arch_os_abi,
691 .cpu_features = cpu_features,
692 }) catch unreachable) catch unreachable),
693 .output_mode = .Exe,
694 });
695 return base_path.join(arena, bin_name);
696}
697
698pub fn updateTimeReportCompile(ws: *WebServer, opts: struct {
699 compile: *Build.Step.Compile,
700
701 use_llvm: bool,
702 stats: abi.time_report.CompileResult.Stats,
703 ns_total: u64,
704
705 llvm_pass_timings_len: u32,
706 files_len: u32,
707 decls_len: u32,
708
709 /// The trailing data of `abi.time_report.CompileResult`, except the step name.
710 trailing: []const u8,
711}) void {
712 const gpa = ws.gpa;
713
714 const step_idx: u32 = for (ws.all_steps, 0..) |s, i| {
715 if (s == &opts.compile.step) break @intCast(i);
716 } else unreachable;
717
718 const old_buf = old: {
719 ws.time_report_mutex.lock();
720 defer ws.time_report_mutex.unlock();
721 const old = ws.time_report_msgs[step_idx];
722 ws.time_report_msgs[step_idx] = &.{};
723 break :old old;
724 };
725 const buf = gpa.realloc(old_buf, @sizeOf(abi.time_report.CompileResult) + opts.trailing.len) catch @panic("out of memory");
726
727 const out_header: *align(1) abi.time_report.CompileResult = @ptrCast(buf[0..@sizeOf(abi.time_report.CompileResult)]);
728 out_header.* = .{
729 .step_idx = step_idx,
730 .flags = .{
731 .use_llvm = opts.use_llvm,
732 },
733 .stats = opts.stats,
734 .ns_total = opts.ns_total,
735 .llvm_pass_timings_len = opts.llvm_pass_timings_len,
736 .files_len = opts.files_len,
737 .decls_len = opts.decls_len,
738 };
739 @memcpy(buf[@sizeOf(abi.time_report.CompileResult)..], opts.trailing);
740
741 {
742 ws.time_report_mutex.lock();
743 defer ws.time_report_mutex.unlock();
744 assert(ws.time_report_msgs[step_idx].len == 0);
745 ws.time_report_msgs[step_idx] = buf;
746 ws.time_report_update_times[step_idx] = ws.now();
747 }
748 ws.notifyUpdate();
749}
750
751pub fn updateTimeReportGeneric(ws: *WebServer, step: *Build.Step, ns_total: u64) void {
752 const gpa = ws.gpa;
753
754 const step_idx: u32 = for (ws.all_steps, 0..) |s, i| {
755 if (s == step) break @intCast(i);
756 } else unreachable;
757
758 const old_buf = old: {
759 ws.time_report_mutex.lock();
760 defer ws.time_report_mutex.unlock();
761 const old = ws.time_report_msgs[step_idx];
762 ws.time_report_msgs[step_idx] = &.{};
763 break :old old;
764 };
765 const buf = gpa.realloc(old_buf, @sizeOf(abi.time_report.GenericResult)) catch @panic("out of memory");
766 const out: *align(1) abi.time_report.GenericResult = @ptrCast(buf);
767 out.* = .{
768 .step_idx = step_idx,
769 .ns_total = ns_total,
770 };
771 {
772 ws.time_report_mutex.lock();
773 defer ws.time_report_mutex.unlock();
774 assert(ws.time_report_msgs[step_idx].len == 0);
775 ws.time_report_msgs[step_idx] = buf;
776 ws.time_report_update_times[step_idx] = ws.now();
777 }
778 ws.notifyUpdate();
779}
780
781const RunnerRequest = union(enum) {
782 rebuild,
783};
784pub fn getRunnerRequest(ws: *WebServer) ?RunnerRequest {
785 ws.runner_request_mutex.lock();
786 defer ws.runner_request_mutex.unlock();
787 if (ws.runner_request) |req| {
788 ws.runner_request = null;
789 ws.runner_request_empty_cond.signal();
790 return req;
791 }
792 return null;
793}
794pub fn wait(ws: *WebServer) RunnerRequest {
795 ws.runner_request_mutex.lock();
796 defer ws.runner_request_mutex.unlock();
797 while (true) {
798 if (ws.runner_request) |req| {
799 ws.runner_request = null;
800 ws.runner_request_empty_cond.signal();
801 return req;
802 }
803 ws.runner_request_ready_cond.wait(&ws.runner_request_mutex);
804 }
805}
806
807const cache_control_header: std.http.Header = .{
808 .name = "Cache-Control",
809 .value = "max-age=0, must-revalidate",
810};
811
812const builtin = @import("builtin");
813const std = @import("std");
814const assert = std.debug.assert;
815const mem = std.mem;
816const log = std.log.scoped(.web_server);
817const Allocator = std.mem.Allocator;
818const Build = std.Build;
819const Cache = Build.Cache;
820const Fuzz = Build.Fuzz;
821const abi = Build.abi;
822
823const WebServer = @This();
lib/std/Build/abi.zig created+313
......@@ -0,0 +1,313 @@
1//! This file is shared among Zig code running in wildly different contexts:
2//! * The build runner, running on the host computer
3//! * The build system web interface Wasm code, running in the browser
4//! * `libfuzzer`, compiled alongside unit tests
5//!
6//! All of these components interface to some degree via an ABI:
7//! * The build runner communicates with the web interface over a WebSocket connection
8//! * The build runner communicates with `libfuzzer` over a shared memory-mapped file
9
10// Check that no WebSocket message type has implicit padding bits. This ensures we never send any
11// undefined bits over the wire, and also helps validate that the layout doesn't differ between, for
12// instance, the web server in `std.Build` and the Wasm client.
13comptime {
14 const check = struct {
15 fn check(comptime T: type) void {
16 const std = @import("std");
17 std.debug.assert(@typeInfo(T) == .@"struct");
18 std.debug.assert(@typeInfo(T).@"struct".layout == .@"extern");
19 std.debug.assert(std.meta.hasUniqueRepresentation(T));
20 }
21 }.check;
22
23 // server->client
24 check(Hello);
25 check(StatusUpdate);
26 check(StepUpdate);
27 check(fuzz.SourceIndexHeader);
28 check(fuzz.CoverageUpdateHeader);
29 check(fuzz.EntryPointHeader);
30 check(time_report.GenericResult);
31 check(time_report.CompileResult);
32
33 // client->server
34 check(Rebuild);
35}
36
37/// All WebSocket messages sent by the server to the client begin with a `ToClientTag` byte. This
38/// enum is non-exhaustive only to avoid Illegal Behavior when malformed messages are sent over the
39/// socket; unnamed tags are an error condition and should terminate the connection.
40///
41/// Every tag has a curresponding `extern struct` representing the full message (or a header of the
42/// message if it is variable-length). For instance, `.hello` corresponds to `Hello`.
43///
44/// When introducing a tag, make sure to add a corresponding `extern struct` whose first field is
45/// this enum, and `check` its layout in the `comptime` block above.
46pub const ToClientTag = enum(u8) {
47 hello,
48 status_update,
49 step_update,
50
51 // `--fuzz`
52 fuzz_source_index,
53 fuzz_coverage_update,
54 fuzz_entry_points,
55
56 // `--time-report`
57 time_report_generic_result,
58 time_report_compile_result,
59
60 _,
61};
62
63/// Like `ToClientTag`, but for messages sent by the client to the server.
64pub const ToServerTag = enum(u8) {
65 rebuild,
66
67 _,
68};
69
70/// The current overall status of the build runner.
71/// Keep in sync with indices in web UI `main.js:updateBuildStatus`.
72pub const BuildStatus = enum(u8) {
73 idle,
74 watching,
75 running,
76 fuzz_init,
77};
78
79/// WebSocket server->client.
80///
81/// Sent by the server as the first message after a WebSocket connection opens to provide basic
82/// information about the server, the build graph, etc.
83///
84/// Trailing:
85/// * `step_name_len: u32` for each `steps_len`
86/// * `step_name: [step_name_len]u8` for each `step_name_len`
87/// * `step_status: u8` for every 4 `steps_len`; every 2 bits is a `StepUpdate.Status`, LSBs first
88pub const Hello = extern struct {
89 tag: ToClientTag = .hello,
90
91 status: BuildStatus,
92 flags: Flags,
93
94 /// Any message containing a timestamp represents it as a number of nanoseconds relative to when
95 /// the build began. This field is the current timestamp, represented in that form.
96 timestamp: i64 align(4),
97
98 /// The number of steps in the build graph which are reachable from the top-level step[s] being
99 /// run; in other words, the number of steps which will be executed by this build. The name of
100 /// each step trails this message.
101 steps_len: u32 align(1),
102
103 pub const Flags = packed struct(u16) {
104 /// Whether time reporting is enabled.
105 time_report: bool,
106 _: u15 = 0,
107 };
108};
109/// WebSocket server->client.
110///
111/// Indicates that the build status has changed.
112pub const StatusUpdate = extern struct {
113 tag: ToClientTag = .status_update,
114 new: BuildStatus,
115};
116/// WebSocket server->client.
117///
118/// Indicates a change in a step's status.
119pub const StepUpdate = extern struct {
120 tag: ToClientTag = .step_update,
121 step_idx: u32 align(1),
122 bits: packed struct(u8) {
123 status: Status,
124 _: u6 = 0,
125 },
126 /// Keep in sync with indices in web UI `main.js:updateStepStatus`.
127 pub const Status = enum(u2) {
128 pending,
129 wip,
130 success,
131 failure,
132 };
133};
134
135pub const Rebuild = extern struct {
136 tag: ToServerTag = .rebuild,
137};
138
139/// ABI bits specifically relating to the fuzzer interface.
140pub const fuzz = struct {
141 /// libfuzzer uses this and its usize is the one that counts. To match the ABI,
142 /// make the ints be the size of the target used with libfuzzer.
143 ///
144 /// Trailing:
145 /// * 1 bit per pc_addr, usize elements
146 /// * pc_addr: usize for each pcs_len
147 pub const SeenPcsHeader = extern struct {
148 n_runs: usize,
149 unique_runs: usize,
150 pcs_len: usize,
151
152 /// Used for comptime assertions. Provides a mechanism for strategically
153 /// causing compile errors.
154 pub const trailing = .{
155 .pc_bits_usize,
156 .pc_addr,
157 };
158
159 pub fn headerEnd(header: *const SeenPcsHeader) []const usize {
160 const ptr: [*]align(@alignOf(usize)) const u8 = @ptrCast(header);
161 const header_end_ptr: [*]const usize = @ptrCast(ptr + @sizeOf(SeenPcsHeader));
162 const pcs_len = header.pcs_len;
163 return header_end_ptr[0 .. pcs_len + seenElemsLen(pcs_len)];
164 }
165
166 pub fn seenBits(header: *const SeenPcsHeader) []const usize {
167 return header.headerEnd()[0..seenElemsLen(header.pcs_len)];
168 }
169
170 pub fn seenElemsLen(pcs_len: usize) usize {
171 return (pcs_len + @bitSizeOf(usize) - 1) / @bitSizeOf(usize);
172 }
173
174 pub fn pcAddrs(header: *const SeenPcsHeader) []const usize {
175 const pcs_len = header.pcs_len;
176 return header.headerEnd()[seenElemsLen(pcs_len)..][0..pcs_len];
177 }
178 };
179
180 /// WebSocket server->client.
181 ///
182 /// Sent once, when fuzzing starts, to indicate the available coverage data.
183 ///
184 /// Trailing:
185 /// * std.debug.Coverage.String for each directories_len
186 /// * std.debug.Coverage.File for each files_len
187 /// * std.debug.Coverage.SourceLocation for each source_locations_len
188 /// * u8 for each string_bytes_len
189 pub const SourceIndexHeader = extern struct {
190 tag: ToClientTag = .fuzz_source_index,
191 _: [3]u8 = @splat(0),
192 directories_len: u32,
193 files_len: u32,
194 source_locations_len: u32,
195 string_bytes_len: u32,
196 /// When, according to the server, fuzzing started.
197 start_timestamp: i64 align(4),
198 };
199
200 /// WebSocket server->client.
201 ///
202 /// Sent whenever the set of covered source locations is updated.
203 ///
204 /// Trailing:
205 /// * one bit per source_locations_len, contained in u64 elements
206 pub const CoverageUpdateHeader = extern struct {
207 tag: ToClientTag = .fuzz_coverage_update,
208 _: [7]u8 = @splat(0),
209 n_runs: u64,
210 unique_runs: u64,
211
212 pub const trailing = .{
213 .pc_bits_usize,
214 };
215 };
216
217 /// WebSocket server->client.
218 ///
219 /// Sent whenever the set of entry points is updated.
220 ///
221 /// Trailing:
222 /// * one u32 index of source_locations per locsLen()
223 pub const EntryPointHeader = extern struct {
224 tag: ToClientTag = .fuzz_entry_points,
225 locs_len_raw: [3]u8,
226
227 pub fn locsLen(hdr: EntryPointHeader) u24 {
228 return @bitCast(hdr.locs_len_raw);
229 }
230 pub fn init(locs_len: u24) EntryPointHeader {
231 return .{ .locs_len_raw = @bitCast(locs_len) };
232 }
233 };
234};
235
236/// ABI bits specifically relating to the time report interface.
237pub const time_report = struct {
238 /// WebSocket server->client.
239 ///
240 /// Sent after a `Step` finishes, providing the time taken to execute the step.
241 pub const GenericResult = extern struct {
242 tag: ToClientTag = .time_report_generic_result,
243 step_idx: u32 align(1),
244 ns_total: u64 align(1),
245 };
246
247 /// WebSocket server->client.
248 ///
249 /// Sent after a `Step.Compile` finishes, providing the step's time report.
250 ///
251 /// Trailing:
252 /// * `llvm_pass_timings: [llvm_pass_timings_len]u8` (ASCII-encoded)
253 /// * for each `files_len`:
254 /// * `name` (null-terminated UTF-8 string)
255 /// * for each `decls_len`:
256 /// * `name` (null-terminated UTF-8 string)
257 /// * `file: u32` (index of file this decl is in)
258 /// * `sema_ns: u64` (nanoseconds spent semantically analyzing this decl)
259 /// * `codegen_ns: u64` (nanoseconds spent semantically analyzing this decl)
260 /// * `link_ns: u64` (nanoseconds spent semantically analyzing this decl)
261 pub const CompileResult = extern struct {
262 tag: ToClientTag = .time_report_compile_result,
263
264 step_idx: u32 align(1),
265
266 flags: Flags,
267 stats: Stats align(1),
268 ns_total: u64 align(1),
269
270 llvm_pass_timings_len: u32 align(1),
271 files_len: u32 align(1),
272 decls_len: u32 align(1),
273
274 pub const Flags = packed struct(u8) {
275 use_llvm: bool,
276 _: u7 = 0,
277 };
278
279 pub const Stats = extern struct {
280 n_reachable_files: u32,
281 n_imported_files: u32,
282 n_generic_instances: u32,
283 n_inline_calls: u32,
284
285 cpu_ns_parse: u64,
286 cpu_ns_astgen: u64,
287 cpu_ns_sema: u64,
288 cpu_ns_codegen: u64,
289 cpu_ns_link: u64,
290
291 real_ns_files: u64,
292 real_ns_decls: u64,
293 real_ns_llvm_emit: u64,
294 real_ns_link_flush: u64,
295
296 pub const init: Stats = .{
297 .n_reachable_files = 0,
298 .n_imported_files = 0,
299 .n_generic_instances = 0,
300 .n_inline_calls = 0,
301 .cpu_ns_parse = 0,
302 .cpu_ns_astgen = 0,
303 .cpu_ns_sema = 0,
304 .cpu_ns_codegen = 0,
305 .cpu_ns_link = 0,
306 .real_ns_files = 0,
307 .real_ns_decls = 0,
308 .real_ns_llvm_emit = 0,
309 .real_ns_link_flush = 0,
310 };
311 };
312 };
313};
lib/std/http/WebSocket.zig+6-8
......@@ -18,14 +18,13 @@ pub const InitError = error{WebSocketUpgradeMissingKey} ||
1818 std.http.Server.Request.ReaderError;
1919
2020pub fn init(
21 ws: *WebSocket,
2221 request: *std.http.Server.Request,
2322 send_buffer: []u8,
2423 recv_buffer: []align(4) u8,
25) InitError!bool {
24) InitError!?WebSocket {
2625 switch (request.head.version) {
27 .@"HTTP/1.0" => return false,
28 .@"HTTP/1.1" => if (request.head.method != .GET) return false,
26 .@"HTTP/1.0" => return null,
27 .@"HTTP/1.1" => if (request.head.method != .GET) return null,
2928 }
3029
3130 var sec_websocket_key: ?[]const u8 = null;
......@@ -36,12 +35,12 @@ pub fn init(
3635 sec_websocket_key = header.value;
3736 } else if (std.ascii.eqlIgnoreCase(header.name, "upgrade")) {
3837 if (!std.ascii.eqlIgnoreCase(header.value, "websocket"))
39 return false;
38 return null;
4039 upgrade_websocket = true;
4140 }
4241 }
4342 if (!upgrade_websocket)
44 return false;
43 return null;
4544
4645 const key = sec_websocket_key orelse return error.WebSocketUpgradeMissingKey;
4746
......@@ -55,7 +54,7 @@ pub fn init(
5554
5655 request.head.content_length = std.math.maxInt(u64);
5756
58 ws.* = .{
57 return .{
5958 .key = key,
6059 .recv_fifo = std.fifo.LinearFifo(u8, .Slice).init(recv_buffer),
6160 .reader = try request.reader(),
......@@ -74,7 +73,6 @@ pub fn init(
7473 .request = request,
7574 .outstanding_len = 0,
7675 };
77 return true;
7876}
7977
8078pub const Header0 = packed struct(u8) {
lib/std/net.zig+41
......@@ -42,6 +42,47 @@ pub const Address = extern union {
4242 in6: Ip6Address,
4343 un: if (has_unix_sockets) posix.sockaddr.un else void,
4444
45 /// Parse an IP address which may include a port. For IPv4, this is just written `address:port`.
46 /// For IPv6, RFC 3986 defines this as an "IP literal", and the port is differentiated from the
47 /// address by surrounding the address part in brackets '[addr]:port'. Even if the port is not
48 /// given, the brackets are mandatory.
49 pub fn parseIpAndPort(str: []const u8) error{ InvalidAddress, InvalidPort }!Address {
50 if (str.len == 0) return error.InvalidAddress;
51 if (str[0] == '[') {
52 const addr_end = std.mem.indexOfScalar(u8, str, ']') orelse
53 return error.InvalidAddress;
54 const addr_str = str[1..addr_end];
55 const port: u16 = p: {
56 if (addr_end == str.len - 1) break :p 0;
57 if (str[addr_end + 1] != ':') return error.InvalidAddress;
58 break :p parsePort(str[addr_end + 2 ..]) orelse return error.InvalidPort;
59 };
60 return parseIp6(addr_str, port) catch error.InvalidAddress;
61 } else {
62 if (std.mem.indexOfScalar(u8, str, ':')) |idx| {
63 // hold off on `error.InvalidPort` since `error.InvalidAddress` might make more sense
64 const port: ?u16 = parsePort(str[idx + 1 ..]);
65 const addr = parseIp4(str[0..idx], port orelse 0) catch return error.InvalidAddress;
66 if (port == null) return error.InvalidPort;
67 return addr;
68 } else {
69 return parseIp4(str, 0) catch error.InvalidAddress;
70 }
71 }
72 }
73 fn parsePort(str: []const u8) ?u16 {
74 var p: u16 = 0;
75 for (str) |c| switch (c) {
76 '0'...'9' => {
77 const shifted = std.math.mul(u16, p, 10) catch return null;
78 p = std.math.add(u16, shifted, c - '0') catch return null;
79 },
80 else => return null,
81 };
82 if (p == 0) return null;
83 return p;
84 }
85
4586 /// Parse the given IP address string into an Address value.
4687 /// It is recommended to use `resolveIp` instead, to handle
4788 /// IPv6 link-local unix addresses.
lib/std/zig/Server.zig+15
......@@ -50,6 +50,8 @@ pub const Message = struct {
5050 /// address of the fuzz unit test. This is used to provide a starting
5151 /// point to view coverage.
5252 fuzz_start_addr,
53 /// Body is a TimeReport.
54 time_report,
5355
5456 _,
5557 };
......@@ -95,6 +97,19 @@ pub const Message = struct {
9597 };
9698 };
9799
100 /// Trailing is the same as in `std.Build.abi.time_report.CompileResult`, excluding `step_name`.
101 pub const TimeReport = extern struct {
102 stats: std.Build.abi.time_report.CompileResult.Stats align(4),
103 llvm_pass_timings_len: u32,
104 files_len: u32,
105 decls_len: u32,
106 flags: Flags,
107 pub const Flags = packed struct(u32) {
108 use_llvm: bool,
109 _: u31 = 0,
110 };
111 };
112
98113 /// Trailing:
99114 /// * the hex digest of the cache directory within the /o/ subdirectory.
100115 pub const EmitDigest = extern struct {
src/Compilation.zig+191-5
......@@ -173,7 +173,6 @@ verbose_cimport: bool,
173173verbose_llvm_cpu_features: bool,
174174verbose_link: bool,
175175disable_c_depfile: bool,
176time_report: bool,
177176stack_report: bool,
178177debug_compiler_runtime_libs: bool,
179178debug_compile_errors: bool,
......@@ -263,6 +262,8 @@ link_prog_node: std.Progress.Node = std.Progress.Node.none,
263262
264263llvm_opt_bisect_limit: c_int,
265264
265time_report: ?TimeReport,
266
266267file_system_inputs: ?*std.ArrayListUnmanaged(u8),
267268
268269/// This is the digest of the cache for the current compilation.
......@@ -322,6 +323,72 @@ const QueuedJobs = struct {
322323 zigc_lib: bool = false,
323324};
324325
326pub const Timer = union(enum) {
327 unused,
328 active: struct {
329 start: std.time.Instant,
330 saved_ns: u64,
331 },
332 paused: u64,
333 stopped,
334
335 pub fn pause(t: *Timer) void {
336 switch (t.*) {
337 .unused => return,
338 .active => |a| {
339 const current = std.time.Instant.now() catch unreachable;
340 const new_ns = switch (current.order(a.start)) {
341 .lt, .eq => 0,
342 .gt => current.since(a.start),
343 };
344 t.* = .{ .paused = a.saved_ns + new_ns };
345 },
346 .paused => unreachable,
347 .stopped => unreachable,
348 }
349 }
350 pub fn @"resume"(t: *Timer) void {
351 switch (t.*) {
352 .unused => return,
353 .active => unreachable,
354 .paused => |saved_ns| t.* = .{ .active = .{
355 .start = std.time.Instant.now() catch unreachable,
356 .saved_ns = saved_ns,
357 } },
358 .stopped => unreachable,
359 }
360 }
361 pub fn finish(t: *Timer) ?u64 {
362 defer t.* = .stopped;
363 switch (t.*) {
364 .unused => return null,
365 .active => |a| {
366 const current = std.time.Instant.now() catch unreachable;
367 const new_ns = switch (current.order(a.start)) {
368 .lt, .eq => 0,
369 .gt => current.since(a.start),
370 };
371 return a.saved_ns + new_ns;
372 },
373 .paused => |ns| return ns,
374 .stopped => unreachable,
375 }
376 }
377};
378
379/// Starts a timer for measuring a `--time-report` value. If `comp.time_report` is `null`, the
380/// returned timer does nothing. When the thing being timed is done, call `Timer.finish`. If that
381/// function returns non-`null`, then the value is a number of nanoseconds, and `comp.time_report`
382/// is set.
383pub fn startTimer(comp: *Compilation) Timer {
384 if (comp.time_report == null) return .unused;
385 const now = std.time.Instant.now() catch @panic("std.time.Timer unsupported; cannot emit time report");
386 return .{ .active = .{
387 .start = now,
388 .saved_ns = 0,
389 } };
390}
391
325392/// A filesystem path, represented relative to one of a few specific directories where possible.
326393/// Every path (considering symlinks as distinct paths) has a canonical representation in this form.
327394/// This abstraction allows us to:
......@@ -787,6 +854,58 @@ pub inline fn debugIncremental(comp: *const Compilation) bool {
787854 return comp.debug_incremental;
788855}
789856
857pub const TimeReport = struct {
858 stats: std.Build.abi.time_report.CompileResult.Stats,
859
860 /// Allocated into `gpa`. The pass time statistics emitted by LLVM's "time-passes" option.
861 /// LLVM provides this data in ASCII form as a table, which can be directly shown to users.
862 ///
863 /// Ideally, we would be able to use `printAllJSONValues` to get *structured* data which we can
864 /// then display more nicely. Unfortunately, that function seems to trip an assertion on one of
865 /// the pass timer names at the time of writing.
866 llvm_pass_timings: []u8,
867
868 /// Key is a ZIR `declaration` instruction; value is the number of nanoseconds spent analyzing
869 /// it. This is the total across all instances of the generic parent namespace, and (if this is
870 /// a function) all generic instances of this function. It also includes time spent analyzing
871 /// function bodies if this is a function (generic or otherwise).
872 /// An entry not existing means the declaration has not been analyzed (so far).
873 decl_sema_info: std.AutoArrayHashMapUnmanaged(InternPool.TrackedInst.Index, struct {
874 ns: u64,
875 count: u32,
876 }),
877
878 /// Key is a ZIR `declaration` instruction which is a function or test; value is the number of
879 /// nanoseconds spent running codegen on it. As above, this is the total across all generic
880 /// instances, both of this function itself and of its parent namespace.
881 /// An entry not existing means the declaration has not been codegenned (so far).
882 /// Every key in `decl_codegen_ns` is also in `decl_sema_ns`.
883 decl_codegen_ns: std.AutoArrayHashMapUnmanaged(InternPool.TrackedInst.Index, u64),
884
885 /// Key is a ZIR `declaration` instruction which is anything other than a `comptime` decl; value
886 /// is the number of nanoseconds spent linking it into the binary. As above, this is the total
887 /// across all generic instances.
888 /// An entry not existing means the declaration has not been linked (so far).
889 /// Every key in `decl_link_ns` is also in `decl_sema_ns`.
890 decl_link_ns: std.AutoArrayHashMapUnmanaged(InternPool.TrackedInst.Index, u64),
891
892 pub fn deinit(tr: *TimeReport, gpa: Allocator) void {
893 tr.stats = undefined;
894 gpa.free(tr.llvm_pass_timings);
895 tr.decl_sema_info.deinit(gpa);
896 tr.decl_codegen_ns.deinit(gpa);
897 tr.decl_link_ns.deinit(gpa);
898 }
899
900 pub const init: TimeReport = .{
901 .stats = .init,
902 .llvm_pass_timings = &.{},
903 .decl_sema_info = .empty,
904 .decl_codegen_ns = .empty,
905 .decl_link_ns = .empty,
906 };
907};
908
790909pub const default_stack_protector_buffer_size = target_util.default_stack_protector_buffer_size;
791910pub const SemaError = Zcu.SemaError;
792911
......@@ -2027,7 +2146,7 @@ pub fn create(gpa: Allocator, arena: Allocator, options: CreateOptions) !*Compil
20272146 .verbose_link = options.verbose_link,
20282147 .disable_c_depfile = options.disable_c_depfile,
20292148 .reference_trace = options.reference_trace,
2030 .time_report = options.time_report,
2149 .time_report = if (options.time_report) .init else null,
20312150 .stack_report = options.stack_report,
20322151 .test_filters = options.test_filters,
20332152 .test_name_prefix = options.test_name_prefix,
......@@ -2561,6 +2680,8 @@ pub fn destroy(comp: *Compilation) void {
25612680 }
25622681 comp.failed_win32_resources.deinit(gpa);
25632682
2683 if (comp.time_report) |*tr| tr.deinit(gpa);
2684
25642685 comp.link_diags.deinit();
25652686
25662687 comp.clearMiscFailures();
......@@ -2657,6 +2778,10 @@ pub fn update(comp: *Compilation, main_progress_node: std.Progress.Node) !void {
26572778
26582779 comp.clearMiscFailures();
26592780 comp.last_update_was_cache_hit = false;
2781 if (comp.time_report) |*tr| {
2782 tr.deinit(gpa); // this is information about an old update
2783 tr.* = .init;
2784 }
26602785
26612786 var tmp_dir_rand_int: u64 = undefined;
26622787 var man: Cache.Manifest = undefined;
......@@ -2688,6 +2813,14 @@ pub fn update(comp: *Compilation, main_progress_node: std.Progress.Node) !void {
26882813 whole.cache_manifest = &man;
26892814 try addNonIncrementalStuffToCacheManifest(comp, arena, &man);
26902815
2816 // Under `--time-report`, ignore cache hits; do the work anyway for those juicy numbers.
2817 const ignore_hit = comp.time_report != null;
2818
2819 if (ignore_hit) {
2820 // We're going to do the work regardless of whether this is a hit or a miss.
2821 man.want_shared_lock = false;
2822 }
2823
26912824 const is_hit = man.hit() catch |err| switch (err) {
26922825 error.CacheCheckFailed => switch (man.diagnostic) {
26932826 .none => unreachable,
......@@ -2713,7 +2846,7 @@ pub fn update(comp: *Compilation, main_progress_node: std.Progress.Node) !void {
27132846 .{},
27142847 ),
27152848 };
2716 if (is_hit) {
2849 if (is_hit and !ignore_hit) {
27172850 // In this case the cache hit contains the full set of file system inputs. Nice!
27182851 if (comp.file_system_inputs) |buf| try man.populateFileSystemInputs(buf);
27192852 if (comp.parent_whole_cache) |pwc| {
......@@ -2734,6 +2867,11 @@ pub fn update(comp: *Compilation, main_progress_node: std.Progress.Node) !void {
27342867 }
27352868 log.debug("CacheMode.whole cache miss for {s}", .{comp.root_name});
27362869
2870 if (ignore_hit) {
2871 // Okay, now set this back so that `writeManifest` will downgrade our lock later.
2872 man.want_shared_lock = true;
2873 }
2874
27372875 // Compile the artifacts to a temporary directory.
27382876 whole.tmp_artifact_directory = d: {
27392877 tmp_dir_rand_int = std.crypto.random.int(u64);
......@@ -2786,6 +2924,8 @@ pub fn update(comp: *Compilation, main_progress_node: std.Progress.Node) !void {
27862924 const pt: Zcu.PerThread = .activate(zcu, .main);
27872925 defer pt.deactivate();
27882926
2927 assert(zcu.cur_analysis_timer == null);
2928
27892929 zcu.skip_analysis_this_update = false;
27902930
27912931 // TODO: doing this in `resolveReferences` later could avoid adding inputs for dead embedfiles. Investigate!
......@@ -2829,6 +2969,8 @@ pub fn update(comp: *Compilation, main_progress_node: std.Progress.Node) !void {
28292969 const pt: Zcu.PerThread = .activate(zcu, .main);
28302970 defer pt.deactivate();
28312971
2972 assert(zcu.cur_analysis_timer == null);
2973
28322974 if (!zcu.skip_analysis_this_update) {
28332975 if (comp.config.is_test) {
28342976 // The `test_functions` decl has been intentionally postponed until now,
......@@ -3040,11 +3182,22 @@ fn flush(
30403182) !void {
30413183 if (comp.zcu) |zcu| {
30423184 if (zcu.llvm_object) |llvm_object| {
3185 const pt: Zcu.PerThread = .activate(zcu, tid);
3186 defer pt.deactivate();
3187
30433188 // Emit the ZCU object from LLVM now; it's required to flush the output file.
30443189 // If there's an output file, it wants to decide where the LLVM object goes!
30453190 const sub_prog_node = comp.link_prog_node.start("LLVM Emit Object", 0);
30463191 defer sub_prog_node.end();
3047 try llvm_object.emit(.{ .zcu = zcu, .tid = tid }, .{
3192
3193 var timer = comp.startTimer();
3194 defer if (timer.finish()) |ns| {
3195 comp.mutex.lock();
3196 defer comp.mutex.unlock();
3197 comp.time_report.?.stats.real_ns_llvm_emit = ns;
3198 };
3199
3200 try llvm_object.emit(pt, .{
30483201 .pre_ir_path = comp.verbose_llvm_ir,
30493202 .pre_bc_path = comp.verbose_llvm_bc,
30503203
......@@ -3071,7 +3224,7 @@ fn flush(
30713224
30723225 .is_debug = comp.root_mod.optimize_mode == .Debug,
30733226 .is_small = comp.root_mod.optimize_mode == .ReleaseSmall,
3074 .time_report = comp.time_report,
3227 .time_report = if (comp.time_report) |*p| p else null,
30753228 .sanitize_thread = comp.config.any_sanitize_thread,
30763229 .fuzz = comp.config.any_fuzz,
30773230 .lto = comp.config.lto,
......@@ -3079,6 +3232,12 @@ fn flush(
30793232 }
30803233 }
30813234 if (comp.bin_file) |lf| {
3235 var timer = comp.startTimer();
3236 defer if (timer.finish()) |ns| {
3237 comp.mutex.lock();
3238 defer comp.mutex.unlock();
3239 comp.time_report.?.stats.real_ns_link_flush = ns;
3240 };
30823241 // This is needed before reading the error flags.
30833242 lf.flush(arena, tid, comp.link_prog_node) catch |err| switch (err) {
30843243 error.LinkFailure => {}, // Already reported.
......@@ -4223,6 +4382,17 @@ fn performAllTheWork(
42234382 zcu.generation += 1;
42244383 };
42254384
4385 // This is awkward: we don't want to start the timer until later, but we won't want to stop it
4386 // until the wait groups finish. That means we need do do this.
4387 var decl_work_timer: ?Timer = null;
4388 defer commit_timer: {
4389 const t = &(decl_work_timer orelse break :commit_timer);
4390 const ns = t.finish() orelse break :commit_timer;
4391 comp.mutex.lock();
4392 defer comp.mutex.unlock();
4393 comp.time_report.?.stats.real_ns_decls = ns;
4394 }
4395
42264396 // Here we queue up all the AstGen tasks first, followed by C object compilation.
42274397 // We wait until the AstGen tasks are all completed before proceeding to the
42284398 // (at least for now) single-threaded main work queue. However, C object compilation
......@@ -4431,6 +4601,13 @@ fn performAllTheWork(
44314601 const zir_prog_node = main_progress_node.start("AST Lowering", 0);
44324602 defer zir_prog_node.end();
44334603
4604 var timer = comp.startTimer();
4605 defer if (timer.finish()) |ns| {
4606 comp.mutex.lock();
4607 defer comp.mutex.unlock();
4608 comp.time_report.?.stats.real_ns_files = ns;
4609 };
4610
44344611 var astgen_wait_group: WaitGroup = .{};
44354612 defer astgen_wait_group.wait();
44364613
......@@ -4556,6 +4733,10 @@ fn performAllTheWork(
45564733 return;
45574734 }
45584735
4736 if (comp.time_report) |*tr| {
4737 tr.stats.n_reachable_files = @intCast(zcu.alive_files.count());
4738 }
4739
45594740 if (comp.incremental) {
45604741 const update_zir_refs_node = main_progress_node.start("Update ZIR References", 0);
45614742 defer update_zir_refs_node.end();
......@@ -4599,6 +4780,11 @@ fn performAllTheWork(
45994780 }
46004781 }
46014782
4783 if (comp.zcu != null) {
4784 // Start the timer for the "decls" part of the pipeline (Sema, CodeGen, link).
4785 decl_work_timer = comp.startTimer();
4786 }
4787
46024788 work: while (true) {
46034789 for (&comp.work_queues) |*work_queue| if (work_queue.readItem()) |job| {
46044790 try processOneJob(@intFromEnum(Zcu.PerThread.Id.main), comp, job);
src/Sema.zig+25-15
......@@ -3246,21 +3246,25 @@ fn zirEnumDecl(
32463246 wip_ty.prepare(ip, new_namespace_index);
32473247 done = true;
32483248
3249 try Sema.resolveDeclaredEnum(
3250 pt,
3251 wip_ty,
3252 inst,
3253 tracked_inst,
3254 new_namespace_index,
3255 type_name.name,
3256 small,
3257 body,
3258 tag_type_ref,
3259 any_values,
3260 fields_len,
3261 sema.code,
3262 body_end,
3263 );
3249 {
3250 const tracked_unit = zcu.trackUnitSema(type_name.name.toSlice(ip), null);
3251 defer tracked_unit.end(zcu);
3252 try Sema.resolveDeclaredEnum(
3253 pt,
3254 wip_ty,
3255 inst,
3256 tracked_inst,
3257 new_namespace_index,
3258 type_name.name,
3259 small,
3260 body,
3261 tag_type_ref,
3262 any_values,
3263 fields_len,
3264 sema.code,
3265 body_end,
3266 );
3267 }
32643268
32653269 codegen_type: {
32663270 if (zcu.comp.config.use_llvm) break :codegen_type;
......@@ -7577,6 +7581,12 @@ fn analyzeCall(
75777581
75787582 // This is an inline call. The function must be comptime-known. We will analyze its body directly using this `Sema`.
75797583
7584 if (zcu.comp.time_report) |*tr| {
7585 if (!block.isComptime()) {
7586 tr.stats.n_inline_calls += 1;
7587 }
7588 }
7589
75807590 if (func_ty_info.is_noinline and !block.isComptime()) {
75817591 return sema.fail(block, call_src, "inline call of noinline function", .{});
75827592 }
src/Type.zig+6
......@@ -3797,6 +3797,9 @@ fn resolveStructInner(
37973797 return error.AnalysisFail;
37983798 }
37993799
3800 const tracked_unit = zcu.trackUnitSema(struct_obj.name.toSlice(&zcu.intern_pool), null);
3801 defer tracked_unit.end(zcu);
3802
38003803 if (zcu.comp.debugIncremental()) {
38013804 const info = try zcu.incremental_debug_state.getUnitInfo(gpa, owner);
38023805 info.last_update_gen = zcu.generation;
......@@ -3856,6 +3859,9 @@ fn resolveUnionInner(
38563859 return error.AnalysisFail;
38573860 }
38583861
3862 const tracked_unit = zcu.trackUnitSema(union_obj.name.toSlice(&zcu.intern_pool), null);
3863 defer tracked_unit.end(zcu);
3864
38593865 if (zcu.comp.debugIncremental()) {
38603866 const info = try zcu.incremental_debug_state.getUnitInfo(gpa, owner);
38613867 info.last_update_gen = zcu.generation;
src/Zcu.zig+44-10
......@@ -312,6 +312,10 @@ builtin_decl_values: BuiltinDecl.Memoized = .initFill(.none),
312312incremental_debug_state: if (build_options.enable_debug_extensions) IncrementalDebugState else void =
313313 if (build_options.enable_debug_extensions) .init else {},
314314
315/// Times semantic analysis of the current `AnalUnit`. When we pause to analyze a different unit,
316/// this timer must be temporarily paused and resumed later.
317cur_analysis_timer: ?Compilation.Timer = null,
318
315319generation: u32 = 0,
316320
317321pub const IncrementalDebugState = struct {
......@@ -4683,26 +4687,56 @@ fn explainWhyFileIsInModule(
46834687 }
46844688}
46854689
4686const SemaProgNode = struct {
4690const TrackedUnitSema = struct {
46874691 /// `null` means we created the node, so should end it.
46884692 old_name: ?[std.Progress.Node.max_name_len]u8,
4689 pub fn end(spn: SemaProgNode, zcu: *Zcu) void {
4690 if (spn.old_name) |old_name| {
4693 old_analysis_timer: ?Compilation.Timer,
4694 analysis_timer_decl: ?InternPool.TrackedInst.Index,
4695 pub fn end(tus: TrackedUnitSema, zcu: *Zcu) void {
4696 const comp = zcu.comp;
4697 if (tus.old_name) |old_name| {
46914698 zcu.sema_prog_node.completeOne(); // we're just renaming, but it's effectively completion
46924699 zcu.cur_sema_prog_node.setName(&old_name);
46934700 } else {
46944701 zcu.cur_sema_prog_node.end();
46954702 zcu.cur_sema_prog_node = .none;
46964703 }
4704 report_time: {
4705 const sema_ns = zcu.cur_analysis_timer.?.finish() orelse break :report_time;
4706 const zir_decl = tus.analysis_timer_decl orelse break :report_time;
4707 comp.mutex.lock();
4708 defer comp.mutex.unlock();
4709 comp.time_report.?.stats.cpu_ns_sema += sema_ns;
4710 const gop = comp.time_report.?.decl_sema_info.getOrPut(comp.gpa, zir_decl) catch |err| switch (err) {
4711 error.OutOfMemory => {
4712 comp.setAllocFailure();
4713 break :report_time;
4714 },
4715 };
4716 if (!gop.found_existing) gop.value_ptr.* = .{ .ns = 0, .count = 0 };
4717 gop.value_ptr.ns += sema_ns;
4718 gop.value_ptr.count += 1;
4719 }
4720 zcu.cur_analysis_timer = tus.old_analysis_timer;
4721 if (zcu.cur_analysis_timer) |*t| t.@"resume"();
46974722 }
46984723};
4699pub fn startSemaProgNode(zcu: *Zcu, name: []const u8) SemaProgNode {
4700 if (zcu.cur_sema_prog_node.index != .none) {
4724pub fn trackUnitSema(zcu: *Zcu, name: []const u8, zir_inst: ?InternPool.TrackedInst.Index) TrackedUnitSema {
4725 if (zcu.cur_analysis_timer) |*t| t.pause();
4726 const old_analysis_timer = zcu.cur_analysis_timer;
4727 zcu.cur_analysis_timer = zcu.comp.startTimer();
4728 const old_name: ?[std.Progress.Node.max_name_len]u8 = old_name: {
4729 if (zcu.cur_sema_prog_node.index == .none) {
4730 zcu.cur_sema_prog_node = zcu.sema_prog_node.start(name, 0);
4731 break :old_name null;
4732 }
47014733 const old_name = zcu.cur_sema_prog_node.getName();
47024734 zcu.cur_sema_prog_node.setName(name);
4703 return .{ .old_name = old_name };
4704 } else {
4705 zcu.cur_sema_prog_node = zcu.sema_prog_node.start(name, 0);
4706 return .{ .old_name = null };
4707 }
4735 break :old_name old_name;
4736 };
4737 return .{
4738 .old_name = old_name,
4739 .old_analysis_timer = old_analysis_timer,
4740 .analysis_timer_decl = zir_inst,
4741 };
47084742}
src/Zcu/PerThread.zig+69-9
......@@ -215,12 +215,15 @@ pub fn updateFile(
215215 };
216216 defer cache_file.close();
217217
218 // Under `--time-report`, ignore cache hits; do the work anyway for those juicy numbers.
219 const ignore_hit = comp.time_report != null;
220
218221 const need_update = while (true) {
219222 const result = switch (file.getMode()) {
220223 inline else => |mode| try loadZirZoirCache(zcu, cache_file, stat, file, mode),
221224 };
222225 switch (result) {
223 .success => {
226 .success => if (!ignore_hit) {
224227 log.debug("AstGen cached success: {f}", .{file.path.fmt(comp)});
225228 break false;
226229 },
......@@ -260,9 +263,16 @@ pub fn updateFile(
260263
261264 file.source = source;
262265
266 var timer = comp.startTimer();
263267 // Any potential AST errors are converted to ZIR errors when we run AstGen/ZonGen.
264268 file.tree = try Ast.parse(gpa, source, file.getMode());
269 if (timer.finish()) |ns_parse| {
270 comp.mutex.lock();
271 defer comp.mutex.unlock();
272 comp.time_report.?.stats.cpu_ns_parse += ns_parse;
273 }
265274
275 timer = comp.startTimer();
266276 switch (file.getMode()) {
267277 .zig => {
268278 file.zir = try AstGen.generate(gpa, file.tree.?);
......@@ -282,6 +292,11 @@ pub fn updateFile(
282292 };
283293 },
284294 }
295 if (timer.finish()) |ns_astgen| {
296 comp.mutex.lock();
297 defer comp.mutex.unlock();
298 comp.time_report.?.stats.cpu_ns_astgen += ns_astgen;
299 }
285300
286301 log.debug("AstGen fresh success: {f}", .{file.path.fmt(comp)});
287302 }
......@@ -801,8 +816,11 @@ pub fn ensureComptimeUnitUpToDate(pt: Zcu.PerThread, cu_id: InternPool.ComptimeU
801816 info.deps.clearRetainingCapacity();
802817 }
803818
804 const unit_prog_node = zcu.startSemaProgNode("comptime");
805 defer unit_prog_node.end(zcu);
819 const unit_tracking = zcu.trackUnitSema(
820 "comptime",
821 zcu.intern_pool.getComptimeUnit(cu_id).zir_index,
822 );
823 defer unit_tracking.end(zcu);
806824
807825 return pt.analyzeComptimeUnit(cu_id) catch |err| switch (err) {
808826 error.AnalysisFail => {
......@@ -981,8 +999,8 @@ pub fn ensureNavValUpToDate(pt: Zcu.PerThread, nav_id: InternPool.Nav.Index) Zcu
981999 info.deps.clearRetainingCapacity();
9821000 }
9831001
984 const unit_prog_node = zcu.startSemaProgNode(nav.fqn.toSlice(ip));
985 defer unit_prog_node.end(zcu);
1002 const unit_tracking = zcu.trackUnitSema(nav.fqn.toSlice(ip), nav.srcInst(ip));
1003 defer unit_tracking.end(zcu);
9861004
9871005 const invalidate_value: bool, const new_failed: bool = if (pt.analyzeNavVal(nav_id)) |result| res: {
9881006 break :res .{
......@@ -1381,8 +1399,8 @@ pub fn ensureNavTypeUpToDate(pt: Zcu.PerThread, nav_id: InternPool.Nav.Index) Zc
13811399 info.deps.clearRetainingCapacity();
13821400 }
13831401
1384 const unit_prog_node = zcu.startSemaProgNode(nav.fqn.toSlice(ip));
1385 defer unit_prog_node.end(zcu);
1402 const unit_tracking = zcu.trackUnitSema(nav.fqn.toSlice(ip), nav.srcInst(ip));
1403 defer unit_tracking.end(zcu);
13861404
13871405 const invalidate_type: bool, const new_failed: bool = if (pt.analyzeNavType(nav_id)) |result| res: {
13881406 break :res .{
......@@ -1601,8 +1619,12 @@ pub fn ensureFuncBodyUpToDate(pt: Zcu.PerThread, maybe_coerced_func_index: Inter
16011619 info.deps.clearRetainingCapacity();
16021620 }
16031621
1604 const func_prog_node = zcu.startSemaProgNode(ip.getNav(func.owner_nav).fqn.toSlice(ip));
1605 defer func_prog_node.end(zcu);
1622 const owner_nav = ip.getNav(func.owner_nav);
1623 const unit_tracking = zcu.trackUnitSema(
1624 owner_nav.fqn.toSlice(ip),
1625 owner_nav.srcInst(ip),
1626 );
1627 defer unit_tracking.end(zcu);
16061628
16071629 const ies_outdated, const new_failed = if (pt.analyzeFuncBody(func_index)) |result|
16081630 .{ prev_failed or result.ies_outdated, false }
......@@ -1847,6 +1869,10 @@ fn semaFile(pt: Zcu.PerThread, file_index: Zcu.File.Index) Zcu.SemaError!void {
18471869 });
18481870 const struct_ty = try pt.createFileRootStruct(file_index, new_namespace_index, false);
18491871 errdefer zcu.intern_pool.remove(pt.tid, struct_ty);
1872
1873 if (zcu.comp.time_report) |*tr| {
1874 tr.stats.n_imported_files += 1;
1875 }
18501876}
18511877
18521878/// Called by AstGen worker threads when an import is seen. If `new_file` is returned, the caller is
......@@ -2520,6 +2546,12 @@ pub fn scanNamespace(
25202546 const gpa = zcu.gpa;
25212547 const namespace = zcu.namespacePtr(namespace_index);
25222548
2549 const tracked_unit = zcu.trackUnitSema(
2550 Type.fromInterned(namespace.owner_type).containerTypeName(ip).toSlice(ip),
2551 null,
2552 );
2553 defer tracked_unit.end(zcu);
2554
25232555 // For incremental updates, `scanDecl` wants to look up existing decls by their ZIR index rather
25242556 // than their name. We'll build an efficient mapping now, then discard the current `decls`.
25252557 // We map to the `AnalUnit`, since not every declaration has a `Nav`.
......@@ -2755,6 +2787,12 @@ fn analyzeFnBodyInner(pt: Zcu.PerThread, func_index: InternPool.Index) Zcu.SemaE
27552787 func.setResolvedErrorSet(ip, .none);
27562788 }
27572789
2790 if (zcu.comp.time_report) |*tr| {
2791 if (func.generic_owner != .none) {
2792 tr.stats.n_generic_instances += 1;
2793 }
2794 }
2795
27582796 // This is the `Nau` corresponding to the `declaration` instruction which the function or its generic owner originates from.
27592797 const decl_nav = ip.getNav(if (func.generic_owner == .none)
27602798 func.owner_nav
......@@ -4307,6 +4345,9 @@ pub fn addDependency(pt: Zcu.PerThread, unit: AnalUnit, dependee: InternPool.Dep
43074345/// codegen thread, depending on whether the backend supports `Zcu.Feature.separate_thread`.
43084346pub fn runCodegen(pt: Zcu.PerThread, func_index: InternPool.Index, air: *Air, out: *@import("../link.zig").ZcuTask.LinkFunc.SharedMir) void {
43094347 const zcu = pt.zcu;
4348
4349 var timer = zcu.comp.startTimer();
4350
43104351 const success: bool = if (runCodegenInner(pt, func_index, air)) |mir| success: {
43114352 out.value = mir;
43124353 break :success true;
......@@ -4327,6 +4368,25 @@ pub fn runCodegen(pt: Zcu.PerThread, func_index: InternPool.Index, air: *Air, ou
43274368 }
43284369 break :success false;
43294370 };
4371
4372 if (timer.finish()) |ns_codegen| report_time: {
4373 const ip = &zcu.intern_pool;
4374 const nav = ip.indexToKey(func_index).func.owner_nav;
4375 const zir_decl = ip.getNav(nav).srcInst(ip);
4376 zcu.comp.mutex.lock();
4377 defer zcu.comp.mutex.unlock();
4378 const tr = &zcu.comp.time_report.?;
4379 tr.stats.cpu_ns_codegen += ns_codegen;
4380 const gop = tr.decl_codegen_ns.getOrPut(zcu.gpa, zir_decl) catch |err| switch (err) {
4381 error.OutOfMemory => {
4382 zcu.comp.setAllocFailure();
4383 break :report_time;
4384 },
4385 };
4386 if (!gop.found_existing) gop.value_ptr.* = 0;
4387 gop.value_ptr.* += ns_codegen;
4388 }
4389
43304390 // release `out.value` with this store; synchronizes with acquire loads in `link`
43314391 out.status.store(if (success) .ready else .failed, .release);
43324392 zcu.comp.link_task_queue.mirReady(zcu.comp, func_index, out);
src/codegen/llvm.zig+13-2
......@@ -764,7 +764,7 @@ pub const Object = struct {
764764
765765 is_debug: bool,
766766 is_small: bool,
767 time_report: bool,
767 time_report: ?*Compilation.TimeReport,
768768 sanitize_thread: bool,
769769 fuzz: bool,
770770 lto: std.zig.LtoMode,
......@@ -1063,7 +1063,7 @@ pub const Object = struct {
10631063 var lowered_options: llvm.TargetMachine.EmitOptions = .{
10641064 .is_debug = options.is_debug,
10651065 .is_small = options.is_small,
1066 .time_report = options.time_report,
1066 .time_report_out = null, // set below to make sure it's only set for a single `emitToFile`
10671067 .tsan = options.sanitize_thread,
10681068 .lto = switch (options.lto) {
10691069 .none => .None,
......@@ -1118,6 +1118,11 @@ pub const Object = struct {
11181118 lowered_options.llvm_ir_filename = null;
11191119 }
11201120
1121 var time_report_c_str: [*:0]u8 = undefined;
1122 if (options.time_report != null) {
1123 lowered_options.time_report_out = &time_report_c_str;
1124 }
1125
11211126 lowered_options.asm_filename = options.asm_path;
11221127 if (target_machine.emitToFile(module, &error_message, &lowered_options)) {
11231128 defer llvm.disposeMessage(error_message);
......@@ -1125,6 +1130,12 @@ pub const Object = struct {
11251130 emit_asm_msg, emit_bin_msg, post_llvm_ir_msg, post_llvm_bc_msg, error_message,
11261131 });
11271132 }
1133 if (options.time_report) |tr| {
1134 defer std.c.free(time_report_c_str);
1135 const time_report_data = std.mem.span(time_report_c_str);
1136 assert(tr.llvm_pass_timings.len == 0);
1137 tr.llvm_pass_timings = try comp.gpa.dupe(u8, time_report_data);
1138 }
11281139 }
11291140
11301141 pub fn updateFunc(
src/codegen/llvm/bindings.zig+1-1
......@@ -88,7 +88,7 @@ pub const TargetMachine = opaque {
8888 pub const EmitOptions = extern struct {
8989 is_debug: bool,
9090 is_small: bool,
91 time_report: bool,
91 time_report_out: ?*[*:0]u8,
9292 tsan: bool,
9393 sancov: bool,
9494 lto: LtoPhase,
src/link.zig+33
......@@ -1302,6 +1302,14 @@ pub fn doPrelinkTask(comp: *Compilation, task: PrelinkTask) void {
13021302 comp.link_prog_node.completeOne();
13031303 return;
13041304 };
1305
1306 var timer = comp.startTimer();
1307 defer if (timer.finish()) |ns| {
1308 comp.mutex.lock();
1309 defer comp.mutex.unlock();
1310 comp.time_report.?.stats.cpu_ns_link += ns;
1311 };
1312
13051313 switch (task) {
13061314 .load_explicitly_provided => {
13071315 const prog_node = comp.link_prog_node.start("Parse Inputs", comp.link_inputs.len);
......@@ -1428,6 +1436,9 @@ pub fn doZcuTask(comp: *Compilation, tid: usize, task: ZcuTask) void {
14281436 const ip = &zcu.intern_pool;
14291437 const pt: Zcu.PerThread = .activate(zcu, @enumFromInt(tid));
14301438 defer pt.deactivate();
1439
1440 var timer = comp.startTimer();
1441
14311442 switch (task) {
14321443 .link_nav => |nav_index| {
14331444 const fqn_slice = ip.getNav(nav_index).fqn.toSlice(ip);
......@@ -1502,6 +1513,28 @@ pub fn doZcuTask(comp: *Compilation, tid: usize, task: ZcuTask) void {
15021513 }
15031514 },
15041515 }
1516
1517 if (timer.finish()) |ns_link| report_time: {
1518 const zir_decl: ?InternPool.TrackedInst.Index = switch (task) {
1519 .link_type, .update_line_number => null,
1520 .link_nav => |nav| ip.getNav(nav).srcInst(ip),
1521 .link_func => |f| ip.getNav(ip.indexToKey(f.func).func.owner_nav).srcInst(ip),
1522 };
1523 comp.mutex.lock();
1524 defer comp.mutex.unlock();
1525 const tr = &zcu.comp.time_report.?;
1526 tr.stats.cpu_ns_link += ns_link;
1527 if (zir_decl) |inst| {
1528 const gop = tr.decl_link_ns.getOrPut(zcu.gpa, inst) catch |err| switch (err) {
1529 error.OutOfMemory => {
1530 zcu.comp.setAllocFailure();
1531 break :report_time;
1532 },
1533 };
1534 if (!gop.found_existing) gop.value_ptr.* = 0;
1535 gop.value_ptr.* += ns_link;
1536 }
1537 }
15051538}
15061539/// After the main pipeline is done, but before flush, the compilation may need to link one final
15071540/// `Nav` into the binary: the `builtin.test_functions` value. Since the link thread isn't running
src/main.zig+87-3
......@@ -484,6 +484,7 @@ const usage_build_generic =
484484 \\ -fno-structured-cfg (SPIR-V) force SPIR-V kernels to not use structured control flow
485485 \\ -mexec-model=[value] (WASI) Execution model
486486 \\ -municode (Windows) Use wmain/wWinMain as entry point
487 \\ --time-report Send timing diagnostics to '--listen' clients
487488 \\
488489 \\Per-Module Compile Options:
489490 \\ -target [name] <arch><sub>-<os>-<abi> see the targets command
......@@ -678,7 +679,6 @@ const usage_build_generic =
678679 \\
679680 \\Debug Options (Zig Compiler Development):
680681 \\ -fopt-bisect-limit=[limit] Only run [limit] first LLVM optimization passes
681 \\ -ftime-report Print timing diagnostics
682682 \\ -fstack-report Print stack size diagnostics
683683 \\ --verbose-link Display linker invocations
684684 \\ --verbose-cc Display C compiler invocations
......@@ -1403,7 +1403,7 @@ fn buildOutputType(
14031403 try test_exec_args.append(arena, null);
14041404 } else if (mem.eql(u8, arg, "--test-no-exec")) {
14051405 test_no_exec = true;
1406 } else if (mem.eql(u8, arg, "-ftime-report")) {
1406 } else if (mem.eql(u8, arg, "--time-report")) {
14071407 time_report = true;
14081408 } else if (mem.eql(u8, arg, "-fstack-report")) {
14091409 stack_report = true;
......@@ -2899,6 +2899,10 @@ fn buildOutputType(
28992899 fatal("test-obj requires --test-no-exec", .{});
29002900 }
29012901
2902 if (time_report and listen == .none) {
2903 fatal("--time-report requires --listen", .{});
2904 }
2905
29022906 if (arg_mode == .translate_c and create_module.c_source_files.items.len != 1) {
29032907 fatal("translate-c expects exactly 1 source file (found {d})", .{create_module.c_source_files.items.len});
29042908 }
......@@ -4208,6 +4212,84 @@ fn serveUpdateResults(s: *Server, comp: *Compilation) !void {
42084212 }
42094213 }
42104214
4215 if (comp.time_report) |*tr| {
4216 var decls_len: u32 = 0;
4217
4218 var file_name_bytes: std.ArrayListUnmanaged(u8) = .empty;
4219 defer file_name_bytes.deinit(gpa);
4220 var files: std.AutoArrayHashMapUnmanaged(Zcu.File.Index, void) = .empty;
4221 defer files.deinit(gpa);
4222 var decl_data: std.ArrayListUnmanaged(u8) = .empty;
4223 defer decl_data.deinit(gpa);
4224
4225 // Each decl needs at least 34 bytes:
4226 // * 2 for 1-byte name plus null terminator
4227 // * 4 for `file`
4228 // * 4 for `sema_count`
4229 // * 8 for `sema_ns`
4230 // * 8 for `codegen_ns`
4231 // * 8 for `link_ns`
4232 // Most, if not all, decls in `tr.decl_sema_ns` are valid, so we have a good size estimate.
4233 try decl_data.ensureUnusedCapacity(gpa, tr.decl_sema_info.count() * 34);
4234
4235 for (tr.decl_sema_info.keys(), tr.decl_sema_info.values()) |tracked_inst, sema_info| {
4236 const resolved = tracked_inst.resolveFull(&comp.zcu.?.intern_pool) orelse continue;
4237 const file = comp.zcu.?.fileByIndex(resolved.file);
4238 const zir = file.zir orelse continue;
4239 const decl_name = zir.nullTerminatedString(zir.getDeclaration(resolved.inst).name);
4240
4241 const gop = try files.getOrPut(gpa, resolved.file);
4242 if (!gop.found_existing) try file_name_bytes.writer(gpa).print("{f}\x00", .{file.path.fmt(comp)});
4243
4244 const codegen_ns = tr.decl_codegen_ns.get(tracked_inst) orelse 0;
4245 const link_ns = tr.decl_link_ns.get(tracked_inst) orelse 0;
4246
4247 decls_len += 1;
4248
4249 try decl_data.ensureUnusedCapacity(gpa, 33 + decl_name.len);
4250 decl_data.appendSliceAssumeCapacity(decl_name);
4251 decl_data.appendAssumeCapacity(0);
4252
4253 const out_file = decl_data.addManyAsArrayAssumeCapacity(4);
4254 const out_sema_count = decl_data.addManyAsArrayAssumeCapacity(4);
4255 const out_sema_ns = decl_data.addManyAsArrayAssumeCapacity(8);
4256 const out_codegen_ns = decl_data.addManyAsArrayAssumeCapacity(8);
4257 const out_link_ns = decl_data.addManyAsArrayAssumeCapacity(8);
4258 std.mem.writeInt(u32, out_file, @intCast(gop.index), .little);
4259 std.mem.writeInt(u32, out_sema_count, sema_info.count, .little);
4260 std.mem.writeInt(u64, out_sema_ns, sema_info.ns, .little);
4261 std.mem.writeInt(u64, out_codegen_ns, codegen_ns, .little);
4262 std.mem.writeInt(u64, out_link_ns, link_ns, .little);
4263 }
4264
4265 const header: std.zig.Server.Message.TimeReport = .{
4266 .stats = tr.stats,
4267 .llvm_pass_timings_len = @intCast(tr.llvm_pass_timings.len),
4268 .files_len = @intCast(files.count()),
4269 .decls_len = decls_len,
4270 .flags = .{
4271 .use_llvm = comp.zcu != null and comp.zcu.?.llvm_object != null,
4272 },
4273 };
4274
4275 var slices: [4][]const u8 = .{
4276 @ptrCast(&header),
4277 tr.llvm_pass_timings,
4278 file_name_bytes.items,
4279 decl_data.items,
4280 };
4281 try s.serveMessageHeader(.{
4282 .tag = .time_report,
4283 .bytes_len = len: {
4284 var len: u32 = 0;
4285 for (slices) |slice| len += @intCast(slice.len);
4286 break :len len;
4287 },
4288 });
4289 try s.out.writeVecAll(&slices);
4290 try s.out.flush();
4291 }
4292
42114293 if (error_bundle.errorMessageCount() > 0) {
42124294 try s.serveErrorBundle(error_bundle);
42134295 return;
......@@ -5277,7 +5359,9 @@ fn cmdBuild(gpa: Allocator, arena: Allocator, args: []const []const u8) !void {
52775359 var windows_libs: std.StringArrayHashMapUnmanaged(void) = .empty;
52785360
52795361 if (resolved_target.result.os.tag == .windows) {
5280 try windows_libs.put(arena, "advapi32", {});
5362 try windows_libs.ensureUnusedCapacity(arena, 2);
5363 windows_libs.putAssumeCapacity("advapi32", {});
5364 windows_libs.putAssumeCapacity("ws2_32", {}); // for `--listen` (web interface)
52815365 }
52825366
52835367 const comp = Compilation.create(gpa, arena, .{
src/zig_llvm.cpp+11-4
......@@ -220,7 +220,7 @@ static SanitizerCoverageOptions getSanCovOptions(ZigLLVMCoverageOptions z) {
220220ZIG_EXTERN_C bool ZigLLVMTargetMachineEmitToFile(LLVMTargetMachineRef targ_machine_ref, LLVMModuleRef module_ref,
221221 char **error_message, const ZigLLVMEmitOptions *options)
222222{
223 TimePassesIsEnabled = options->time_report;
223 TimePassesIsEnabled = options->time_report_out != nullptr;
224224
225225 raw_fd_ostream *dest_asm_ptr = nullptr;
226226 raw_fd_ostream *dest_bin_ptr = nullptr;
......@@ -418,10 +418,17 @@ ZIG_EXTERN_C bool ZigLLVMTargetMachineEmitToFile(LLVMTargetMachineRef targ_machi
418418 WriteBitcodeToFile(llvm_module, *dest_bitcode);
419419 }
420420
421 if (options->time_report) {
422 TimerGroup::printAll(errs());
421 // This must only happen once we know we've succeeded and will be returning `false`, because
422 // this code `malloc`s memory which will become owned by the caller (in Zig code).
423 if (options->time_report_out != nullptr) {
424 std::string out_str;
425 auto os = raw_string_ostream(out_str);
426 TimerGroup::printAll(os);
427 TimerGroup::clearAll();
428 auto c_str = (char *)malloc(out_str.length() + 1);
429 strcpy(c_str, out_str.c_str());
430 *options->time_report_out = c_str;
423431 }
424
425432 return false;
426433}
427434
src/zig_llvm.h+4-1
......@@ -66,7 +66,10 @@ enum ZigLLVMThinOrFullLTOPhase {
6666struct ZigLLVMEmitOptions {
6767 bool is_debug;
6868 bool is_small;
69 bool time_report;
69 // If not null, and `ZigLLVMTargetMachineEmitToFile` returns `false` indicating success, this
70 // `char *` will be populated with a `malloc`-allocated string containing the serialized (as
71 // JSON) time report data. The caller is responsible for freeing that memory.
72 char **time_report_out;
7073 bool tsan;
7174 bool sancov;
7275 ZigLLVMThinOrFullLTOPhase lto;
tools/dump-cov.zig+1-1
......@@ -5,7 +5,7 @@ const std = @import("std");
55const fatal = std.process.fatal;
66const Path = std.Build.Cache.Path;
77const assert = std.debug.assert;
8const SeenPcsHeader = std.Build.Fuzz.abi.SeenPcsHeader;
8const SeenPcsHeader = std.Build.abi.fuzz.SeenPcsHeader;
99
1010pub fn main() !void {
1111 var general_purpose_allocator: std.heap.GeneralPurposeAllocator(.{}) = .init;