Mac M2 latest update OS
stuck on Task :metadata-ingestion:runPreFlightScript FAILED
while trying the command ./gradlew :metadata-ingestion:installDev
following the dev guide here https://datahubproject.io/docs/developers/
and my steps
- Make sure you are using the correct java and python, check the versions
1. Which python
2. Python –version
3. Pyenv global 3.10 to change python version
4. Which java
5. Java –version
- Download ARMx53 DRM <https://www.oracle.com/java/technologies/downloads/#jdk17-mac>
1. Make sure java is uninstalled already with `brew remove java`
- export JAVA_HOME=`/usr/libexec/java_home`
- make sure to get the CSP ca cert from company website
- pushd $JAVA_HOME
- cd lib/security
- sudo keytool -importcert -keystore cacerts -storepass changeit -trustcacerts -file ~/Downloads/cspca.cer
- popd
- now that you have the cert try out the datahub commands
- <https://datahubproject.io/docs/developers/>
1. I did not have gradle installed and the gradlew build installed it for me
2. ./gradlew :metadata-service:war:build PASS
3. ./gradlew :datahub-frontend:dist -x yarnTest -x yarnLint PASS
4. ./gradlew :metadata-ingestion:installDev FAIL```
To honour the JVM settings for this build a single-use Daemon process will be forked. See <https://docs.gradle.org/8.0.2/userguide/gradle_daemon.html#sec:disabling_the_daemon>.
Daemon will be stopped at the end of the build
Configuration on demand is an incubating feature.
> Task :li-utils:mainDestroyStaleFiles SKIPPED
:li-utils:mainDestroyStaleFiles task is a NO-OP task.
> Task :li-utils:mainCopyPdscSchemas SKIPPED
:li-utils:mainCopyPdscSchemas task is a NO-OP task.
> Task :metadata-events:mxe-schemas:mainDestroyStaleFiles SKIPPED
:metadata-events:mxe-schemas:mainDestroyStaleFiles task is a NO-OP task.
> Task :metadata-events:mxe-schemas:mainCopyPdscSchemas SKIPPED
:metadata-events:mxe-schemas:mainCopyPdscSchemas task is a NO-OP task.
> Task :metadata-events:mxe-schemas:changedFilesReport
Checking idl and snapshot files for changes...
> Task :metadata-events:mxe-schemas:testDestroyStaleFiles SKIPPED
:metadata-events:mxe-schemas:testDestroyStaleFiles task is a NO-OP task.
> Task :metadata-events:mxe-schemas:testCopyPdscSchemas SKIPPED
:metadata-events:mxe-schemas:testCopyPdscSchemas task is a NO-OP task.
> Task :metadata-ingestion:runPreFlightScript
🔎 Checking if current directory is metadata-ingestion folder
✅ Current folder is metadata-ingestion (/Users/venethongkha1/repos/DataHub/metadata-ingestion) folder
👟 Running preflight for m1 mac
✨ Creating/activating Virtual Environment
🔎 Checking if Scipy installed
Scipy not installed
⛅ Installing prerequisities for scipy NOAH WAS HER==> Downloading <https://ghcr.io/v2/homebrew/core/openblas/manifests/0.3.27>
Already downloaded: /Users/venethongkha1/Library/Caches/Homebrew/downloads/e3edac4ed73cce4aa9b2c78d8a8e22a854ed2d9edb7168c31b864af058ff1a7a--openblas-0.3.27.bottle_manifest.json
==> Fetching dependencies for openblas: gcc
==> Fetching gcc
Warning: Building gcc from source:
The bottle needs the Apple Command Line Tools to be installed.
You can install them, if desired, with:
xcode-select --install
==> Downloading <https://raw.githubusercontent.com/Homebrew/homebrew-core/76f08feae42d2f363b850d6cb2dbaaed554ff9e9/Formula/g/gcc.rb>
Already downloaded: /Users/venethongkha1/Library/Caches/Homebrew/downloads/e74d82df3e8b9bf822a2acb66cd185661be5cbf1aaff6abf688482bebce37867--gcc.rb
==> Downloading <https://raw.githubusercontent.com/Homebrew/formula-patches/3c5cbc8e9cf444a1967786af48e430588e1eb481/gcc/gcc-13.2.0.diff>
Already downloaded: /Users/venethongkha1/Library/Caches/Homebrew/downloads/5463564ab52d0eb177932d8e3bc50285fd1ef2fc0a63c991f3a1287126b29009--gcc-13.2.0.diff
==> Downloading <https://raw.githubusercontent.com/Homebrew/formula-patches/e923a0cd6c0e60bb388e8a5b8cd1dcf9c3bf7758/gcc/gcc-xcode15-warnings.diff>
Already downloaded: /Users/venethongkha1/Library/Caches/Homebrew/downloads/7699750d165d092215d6b9c4adc3d3cae9578cc78de08d8aaae6344303581706--gcc-xcode15-warnings.diff
==> Downloading <https://ftp.gnu.org/gnu/gcc/gcc-13.2.0/gcc-13.2.0.tar.xz>
Already downloaded: /Users/venethongkha1/Library/Caches/Homebrew/downloads/70fae92e139635a60fad468e0d272b2f0099866084a96e38a6645aef6ae21b74--gcc-13.2.0.tar.xz
==> Fetching openblas
==> Downloading <https://ghcr.io/v2/homebrew/core/openblas/blobs/sha256:688a3083556bf2e30e61c25946e964034eeebf6389b9448c2a265fb066da85a4>
Already downloaded: /Users/venethongkha1/Library/Caches/Homebrew/downloads/d19b4208eac80b770bb508dc2c20e3204675dda2ca974245969239df058d637c--openblas--0.3.27.arm64_sonoma.bottle.tar.gz
==> Installing dependencies for openblas: gcc
==> Installing openblas dependency: gcc
xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
Error: An exception occurred within a child process:
CompilerSelectionError: gcc cannot be built with any available compilers.
Install GNU's GCC:
brew install gcc```
using Xcode 15.2 and CLT 15.1 i was able to get past runpreflight but I’m stuck at the 98%
+ uv pip install -e '.[dev]'
Built 1 editable in 2.11s
Resolved 396 packages in 24.11s
error: Failed to download distributions
Caused by: Failed to fetch wheel: pyiceberg==0.0.2
Caused by: Failed to build: pyiceberg==0.0.2
Caused by: Build backend failed to build wheel through `build_wheel()` with exit status: 1
--- stdout:
running bdist_wheel
running build
installing to build/bdist.macosx-14.4-arm64/wheel
running install
--- stderr:
Traceback (most recent call last):
File "<string>", line 11, in <module>
File "/Users/venethongkha1/Library/Caches/uv/.tmpo4v4lU/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 410, in build_wheel
return self._build_with_temp_dir(
File "/Users/venethongkha1/Library/Caches/uv/.tmpo4v4lU/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 395, in _build_with_temp_dir
self.run_setup()
File "/Users/venethongkha1/Library/Caches/uv/.tmpo4v4lU/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 487, in run_setup
super().run_setup(setup_script=setup_script)
File "/Users/venethongkha1/Library/Caches/uv/.tmpo4v4lU/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup
exec(code, locals())
File "<string>", line 10, in <module>
File "/Users/venethongkha1/Library/Caches/uv/.tmpo4v4lU/.venv/lib/python3.10/site-packages/setuptools/__init__.py", line 104, in setup
return distutils.core.setup(**attrs)
File "/Users/venethongkha1/Library/Caches/uv/.tmpo4v4lU/.venv/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 184, in setup
return run_commands(dist)
File "/Users/venethongkha1/Library/Caches/uv/.tmpo4v4lU/.venv/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
dist.run_commands()
File "/Users/venethongkha1/Library/Caches/uv/.tmpo4v4lU/.venv/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/Users/venethongkha1/Library/Caches/uv/.tmpo4v4lU/.venv/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
super().run_command(command)
File "/Users/venethongkha1/Library/Caches/uv/.tmpo4v4lU/.venv/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/Users/venethongkha1/Library/Caches/uv/.tmpo4v4lU/.venv/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 403, in run
self.run_command("install")
File "/Users/venethongkha1/Library/Caches/uv/.tmpo4v4lU/.venv/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
self.distribution.run_command(command)
File "/Users/venethongkha1/Library/Caches/uv/.tmpo4v4lU/.venv/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
super().run_command(command)
File "/Users/venethongkha1/Library/Caches/uv/.tmpo4v4lU/.venv/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "<string>", line 8, in run
RuntimeError: This is a placeholder for the Apache Iceberg Python library
---
> Task :metadata-ingestion:installDev FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':metadata-ingestion:installDev'.
> Process 'command 'bash'' finished with non-zero exit value 2
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at <https://help.gradle.org>
BUILD FAILED in 21m 1s
33 actionable tasks: 12 executed, 1 from cache, 20 up-to-date```
Why do i get this error right here? RuntimeError: This is a placeholder for the Apache Iceberg Python library i tried doing a pip install pyiceberg after and redid my Task :metadata-ingestion:installDev and still failed
Make sure your message includes the following information if relevant, so we can help more effectively!