[dartdev] `dart build cli` support a separate aotruntime mode?

[dcharkes]

We want to introduce build hooks into dart test -c exe (#63372).

However, if we build it on top of dart build cli, we would use dart compile exe under the hood.

This would lose us native profilers and symbolizers support

  /// Compiles [path] to a native shared library using
  /// `dart compile aot-snapshot`.
  ///
  /// Preferable to `dart compile exe` because embedded snapshots are invisible
  /// to native profilers and symbolizers. This should eventually be replaced
  /// with something that supports build hooks.

https://github.com/dart-lang/test/blob/fe9e65866919d28364f2c1bdd806a0a3d5cb101a/pkgs/test_core/lib/src/runner/vm/platform.dart#L248-L253

So, we should consider adding a "mode" to dart build cli that keeps the aot-snapshot separate from the dartaotruntime and generates a shim executable or shell script that makes the bin/my_app run ../bin_private/dartaotruntime ../bin_private/my_aot_snapshot.

Having such a setup would also help with keeping app-bundles with multiple executables smaller, because we'd only include a dartaotruntime in the bundle once.

(N.b. for dart test -c exe it is unnecessary to copy the dartaotruntime, because we have access to a dart sdk, so we can just use that. However, I'd also like to avoid exposing a new dart build subcommand or new dart build cli flags that are only used in package:test. If package:test works as an external package to the SDK it should only rely on well-tested public Dart CLI features. cc @jakemac53 @bkonyi)

@rmacnak-google If you have any other bright ideas, let me know!

[rmacnak-google]

Some brainstorming:

• status quo append snapshot to dartaotruntime executable
  • con: snapshot is invisible to native symbolizers/profilers/debuggers
    • sanitizer errors always come through the native symbolizer!
• full link of snapshot into executable
  • con: complicated
• shell script
  • exec dartaotruntime libapp.so
  • con: no exec on Windows and Fuchsia
    • subprocesses create headaches around killing as we discovered with dartdev
  • con: process name will be either "dartaotruntime" or "dart:libapp.so" instead of "app"
• BusyBox style
  • executable loads snapshot based on argv[0]
  • app1 -> dlopen(libapp1.so)
  • app2 -> dlopen(libapp2.so)
  • myapp1 and myapp2 can be hardlink copies of dartaotruntime
  • con: most ways of distributing the bundle won't preserve the hardlink and will reduplicate
  • con: might be tricky to get symbolic links to the executable to behave as expected

[rmacnak-google]

adding a mode

Should it be a mode? Why keep using dart compile exe for dart build cli? My understanding is the only benefit of dart compile exe is it produces a single file which is easier to distribute. But one starts using dart build cli for asset support and is already getting a bundle instead of a single file.

[dcharkes]

My understanding is the only benefit of dart compile exe is it produces a single file which is easier to distribute.

Because as you noted the shell-script approach has issues too, so it was easier to use the append snapshot to get it shipped as an executable.

• con: no exec on Windows

This is already the case for the installations on PATH on Windows. (For dart run ...@ we do run the executable directly, so that bypasses the shell script.)

sdk/pkg/dartdev/lib/src/install/file_system.dart

Lines 154 to 172 in b69989f

	void createSync(ExecutableInBundle target) {
	if (Platform.isLinux || Platform.isMacOS) {
	return unix.createSync(target.file.path, recursive: true);
	}
	if (Platform.isWindows) {
	final wrapperScriptContents =
	'''
	@ECHO OFF
	REM $_marker
	"${target.file.path}" %*
	EXIT /B %ERRORLEVEL%
	''';
	if (!windows.existsSync()) {
	windows.createSync(recursive: true);
	}
	return windows.writeAsStringSync(wrapperScriptContents);
	}
	throw UnsupportedError('Unsupported OS: ${Platform.operatingSystem}.');
	}

[dcharkes]

If it helps, I can use package:webcrypto as a real test case here. Is there a part of the dart test -c cli --target-sanitizer work you already have planned where I could contribute?

Originally posted by @HamdaanAliQuatil in #278

The first step in that arch of work is this issue: either

1. full link of snapshot into executable for dart compile exe for Linux, or
2. use a BusyBox style packaging in dart build cli on Linux.

My understanding of option 1:

• We need a dartaotruntime build target in the Dart SDK that has one or more external symbols unresolved (and is therefore packaged as a static archive rather than an executable). And we need to distribute that in the SDK.
• We need a native linker accessible in dartdev, which we can do by pulling in native_toolchain_c (it should already be available in the Dart SDK).

The downside is that this option will not work if no C linker is available. (Maybe not super likely, because if you have any build hooks using native toolchain C or if you have your flutter dependencies, you will have a native linker.) We could mitigate this by falling back on the old solution (if no sanitizers are requested).

My understanding of option 2:

• We need a new dartaotruntime with a new main that knows about the bundle layout of dart cli bundle and knows how to find the aotsnapshot dylib relative from the executable. And this needs to be distributed in the SDK.
• the main function basically dlopens the aotsnapshot and then dlsyms the symbols in there.

con: most ways of distributing the bundle won't preserve the hardlink and will reduplicate

I don't think this is so much of a downside, the alternatives will also duplicate.

con: might be tricky to get symbolic links to the executable to behave as expected

I believe we have a bunch of integration tests in dartdev that run dart build cli bundles through symlinks. So that should surface. @rmacnak-google Any scenario specifically that you're worried about?

@rmacnak-google Any more light on the two approaches? Anything complicated in option 1 that I missed?

[rmacnak-google]

For option 2, the new executable might even be able to replace dart-sdk/bin/dart if the setup also works for dartdev.

I think you've covered all the big things to worry about.

[dcharkes]

For option 2, the new executable might even be able to replace dart-sdk/bin/dart if the setup also works for dartdev.

I don't believe so. We don't want to include the JIT compiler in the newdartoatruntime but we need to include it in dart-sdk/bin/dart right?

@HamdaanAliQuatil Let's explore option 2 first and see if we find any downsides with it. Some pointers:

• runtime/bin/main_impl.cc you'll need to wire up a way to take argv[0] and open it instead of argv[1] for the dartaotruntime and compile it to a dartcliruntime
• pkg/dartdev/lib/src/commands/build.dart on Linux needs to have a new dir structure

bundle/
  bin/
    <executable1>
    <executable2>
  lib/
    <dynamic libraries>

->

bundle/
  bin/
    <executable1 = renamed dartcliruntime>
    <executable2 = symlink to executable1>
    <executable3 = symlink to executable1>
  lib/
    dartaotsnapshot<executable1>
    dartaotsnapshot<executable2>
    dartaotsnapshot<executable3>
    <dynamic libraries>

• The dartcliruntime will need to do its dlopen with ../lib/dartaotsnapshot$argv[0].
• All the tests in pkg/dartdev/test/native_assets/build_test.dart should pass tools/test.py -n unittest-asserts-release-linux-x64 pkg/dartdev/test/native_assets/build_test.dart

[HamdaanAliQuatil]

Thanks, that makes sense. I’ll start by exploring option 2 on Linux.

I’ll start by trying the Linux-only dartcliruntime path: have the runtime pick the snapshot from the bundle based on argv[0], then adjust the dart build cli bundle layout and see what breaks in the existing native-assets tests.

I’ll also check whether the snapshot naming/layout should live entirely in dartdev, or whether the runtime needs to know more than just the executable basename.

[mraleph]

but we need to include it in dart-sdk/bin/dart right?

dart is actually an AOT runtime and it can't JIT. That's why it spawns dartvm - which can JIT.