Building and Installing PowerBI Ingestion Integration via Python Package

ingestion
1

Original Slack Thread

<@U06TM7M40RK> How can i build the powerbi ingestion integration into a python package to locally try some changes

2

To build the PowerBI ingestion integration into a Python package for local testing, you can follow these steps:

  1. Clone the DataHub Repository: Start by cloning the DataHub repository to your local machine. This will give you access to the source code for the PowerBI ingestion integration.

    git clone https://github.com/datahub-project/datahub.git
cd datahub

  2. Navigate to the Metadata Ingestion Directory: The PowerBI ingestion code is located within the metadata-ingestion directory. Navigate to this directory to make your changes.

    cd metadata-ingestion

  3. Make Your Code Changes: Modify the PowerBI ingestion source code as needed. This might involve editing files related to the PowerBI ingestion logic.

  4. Build the Python Package: Once your changes are made, you can build the Python package. This involves creating a wheel file that can be installed locally.

    python setup.py bdist_wheel

    This command will generate a .whl file in the dist directory.

  5. Install the Custom Package Locally: You can now install the custom package locally using pip. This allows you to test your changes in a local environment.

    pip install dist/your_custom_package_name.whl

  6. Test Your Changes: With the package installed, you can now test your changes by running the DataHub CLI or any scripts that utilize the PowerBI ingestion integration.

If you need more detailed instructions on any of these steps, please let me know!

Sources:

3

<@U06TM7M40RK> after pip installing and pushing a config via the CLI it says that powerbi is disabled

4

If you encounter a message indicating that PowerBI is disabled after installing the package and pushing a configuration via the CLI, it could be due to several reasons related to configuration or permissions. Here are some steps to troubleshoot and resolve the issue:

  1. Verify PowerBI API and Admin API Settings: Ensure that the PowerBI API and Admin API settings are correctly configured in the PowerBI Admin Portal. According to the <https://datahubproject.io/docs/0.14.0/quick-ingestion-guides/powerbi/setup|PowerBI Ingestion Guide>, you need to enable the following settings:

    • Allow service principals to use Power BI APIs.
    • Allow service principals to use read-only admin APIs.
    • Enhance admin APIs responses with detailed metadata.
    • Enhance admin APIs responses with DAX and mashup expressions.

  2. Check Azure AD App Permissions: Make sure that your Azure AD app has the necessary permissions to access PowerBI resources. This includes being part of a security group with the appropriate permissions as outlined in the <https://datahubproject.io/docs/0.14.0/quick-ingestion-guides/powerbi/setup|PowerBI Setup Guide>.

  3. Review Configuration Parameters: Double-check your configuration parameters to ensure they are correctly set. This includes verifying the client_id, client_secret, and tenant_id in your ingestion recipe. Ensure that these values are not set to None or incorrect.

  4. Service Principal Membership: Ensure that the service principal is added as a member in the workspace you are trying to ingest. This is crucial for the ingestion process to work correctly.

  5. Logs and Error Messages: Review the logs and error messages during the ingestion process. Specific warnings or errors can provide clues about what might be going wrong.

If you need more detailed guidance on any of these steps or additional troubleshooting tips, please let me know!

Sources:

5

<@U06TM7M40RK> how to install power bi ingestion service that i built using python setup.py bdist_wheel

6

To install the PowerBI ingestion service that you built using python setup.py bdist_wheel, follow these steps:

  1. Locate the Wheel File: After running python setup.py bdist_wheel, a .whl file will be generated in the dist directory of your project. This file is your built package.

  2. Install the Wheel File: Use pip to install the wheel file locally. Navigate to the directory containing the .whl file and run the following command:

    pip install dist/your_custom_package_name.whl

    Replace your_custom_package_name.whl with the actual name of the wheel file generated.

  3. Verify Installation: After installation, you can verify that the package is installed by listing the installed packages or checking the version of the DataHub CLI to ensure it recognizes the custom ingestion service.

    pip list | grep your_custom_package_name

    or

    datahub version

  4. Run the Ingestion: With the package installed, you can now run the DataHub CLI to use the PowerBI ingestion service. Ensure that your configuration file is correctly set up and run the ingestion command:

    datahub ingest -c path/to/your/config.yml

If you encounter any issues during installation or need further assistance, feel free to ask for more detailed guidance.

Sources:

  • <Slack Slack Thread> :+1: 0 button :-1: 0 button :bulb: Hint: Mention @runllm in the thread for followups.
7

Hi :slight_smile: ! Did this interaction resolve your issue or do you still require assistance ?

Please tag @ RunLLM if you still need further assistance !

Or tag @Datahub Community Support if you require a person to take a look :slight_smile:

Hope this helps ! ><