set the channel description: Stack Overflow feed

Login time out error for pyodbc connection string in kedro I'm trying to build up connection string in kedro for SQLQueryDataset to set the connection with MSSQL and Pyodbc in Azure Databricks but encountered with an error: (pyodbc.OperationalError) ('HYT00', '[HYT00] [Microsoft][ODBC Driver 17 for SQL Server]Login timeout expired (0) (SQLDriverConnect)') Setting up connection string in credentials.yml file as con: mssql+pyodbc://${sql_username}:${sql_password}@{sql_server_name}/{database_name}?...

"TypeError: Knex is not a function" when configuring Knex with a connection to Supabase (PostgreSQL) i have this problem on configure knex with a connection to supebase (postegresql) Requiring external module ts-node/register Knex is not a function TypeError: Knex is not a function at Object. (C\LAVORO\MagickML\Magick\apps\server\node modules\knex\knex.js32:14) at Module._compile (nodeinternal/modules/cjs/loader1275:14) at Module._extensions..js (nodeinternal/modules/cjs/loader1329:10) at Object.require.extensions. [as .js]...

Where is my kedro output when using the databricks extension in vscode I am using kedro together with the databricks extension for vscode to access databricks server on Azure. Everything works pretty well but I don't see any output when executing the file locally. The only output I receive is: 31/03/2023, 150932 - Synchronizing code to /Repos/@.ide ... 31/03/2023, 150932 - Running kedro-hello-world.py ... [03/31/23 130937] INFO Kedro project dp-kedro session.py:355 31/03/2023, 150943 - Done (took...

TextField covered by BottomBar - Jetpack Compose Hi I have a registration screen with multiple text fields, I use adjustResize and a button in the bottom bar. What to do to make the text field always move above the bottom bar when the keyboard is open or hide bottom bar when the keyboard is open. In a situation where a text field is at the bottom of the screen, when I click on it I can fill it normally, but it is covered by the bottom bar. Scaffold( modifier = Modifier.clearFocus(), topBar = { MainTopBar(...

Kedro Pipeline contains no nodes after applying all provided filters ValueError: Pipeline contains no nodes after applying all provided filters src/pipelines/data_processing src/pipelines/data_science src/pipelines/data_project src/pipelines/pipeline2 Kedro build-docs-> dont'work pip install pip-tools pip install -r src/requirements.txt pip install "kedro-datasets[]" Nothing works [kedro v0.18.7] [Python 3.10.8] My project was working perfectly and at some point I tried to implement Modular Pipelines and then I ran the command to create a new pipeline. [kedro...

kedro: Used azure key and scope, then pull data from snowflake via databricks cluster I'm currently running into a requirement where we are using a scope/key credential to access/grab data from our Snowflake instance, then load it in our blob storage, while using our Databricks cluster. I was already able to do those things in a Databricks notebook, but when porting it into our Kedro framework, it fails. Has anyone done this and can you give an example of a format where you add your azure scope/key and use it in a Kedro catalog? Thanks in advance!

How to call the data object from another namespace in already namespaced kedro pipeline? I have the following namespaced pipelines in kedro: def create_pipeline(**kwargs) -> Pipeline: pipeline_instance = pipeline( [ node( func=split_data, inputs=["model_input_table", "params:model_options"], outputs=["X_train", "X_test", "y_train", "y_test"], name="split_data_node", ), node( func=train_model, inputs=["X_train", "y_train"],...

kedro-mlflow with expiring token I need some help to configure the request_header_provider in mlflow.yml. Today, I have a fully functional Kedro training pipeline for machine learning model, except for the authentication. For the authentication to work, I have to manually: post a request to https://login.microsoftonline.com/xxxxx/oauth2/token with the appropriated payload payload={     "grant_type": "password",    ...

Kedro - how to set max_workers when running pipelines with ParallelRunner? I'm using Kedro version 0.18.7 and python 3.9 in WSL2. I'd like to run nodes of my pipeline in parallel by running the command kedro run --pipeline --runner ParallelRunner. According to the documentation ParallelRunner, it should be possible to define the maximum number of CPU cores to use (using max_workers), but I am struggling to find out how to use this argument. Apparently I cannot...

Unable to Save Data to HDF5 File using fsspec in Python I'm trying to save data (3D numpy array) to an HDF5 file using fsspec in Python, but I'm encountering issues and I am unable to successfully write the data to the file. The bigger picture is I am trying to amend this dataset class to load/write video data. Is it possible to use h5py and fsspec in simultaneously or is it even advised? The root issue seems to be this...

S3FS is not recognised in lambda when using torch I have a Kedro Pipeline Node that on AWS Lambda that accesses s3. It runs if I'm not using torch but fails with Install s3fs to access S3 if I add torch as a dependency. I have a Kedro Pipeline I want to deploy on AWS Step Functions. My requirements look like this: Python3.9 Pillow==9.5.0 aws_lambda_powertools==2.15.0 fsspec==2023.5.0 kedro==0.18.8 numpy==1.24.3 pandas==2.0.1 pydantic==1.10.7 pytest==7.3.1 rasterio==1.3.6 rawpy==0.18.1 s3fs==2023.5.0 The lambda access some data on s3. With...

Access Kedro MemoryDataSet when running packaged Kedro pipeline in a script I want to be able to access a MemoryDataSet results dataframe from a kedro pipeline that I've imported into a script after packaging the pipeline into a python package. I have a kedro pipeline written and run using Kedro=0.18.9 that collects data, modifies it, then runs it against several anomaly detection algorithms. The pipeline loads from a database, saves all intermediate data (preprocessing, etc.) and results to CSVs at every step of the pipeline. It also returns the results of the...

How to combine multiple similar datasets into one in Kedro dynamically? I use Kedro to run pipelines for multiple models that each generate a CSV file with the same schema. However, I need to combine each of the generated CSV files into one and do some post processing which will be my final output. This is what my catalog looks like: catalog.yml model1.output: type: pandas.CSVDataSet filepath: ${data_path}/output/model1/output.csv model2.output: type: pandas.CSVDataSet filepath: ${data_path}/output/model2/output.csv model3.output: type:...

stupid stackoverflow

Is there a way to have files in the Kedro Catalog, that are missing? I have a kedro pipeline which generates a file that is used again for the next run of that same pipeline. However, when the pipeline runs for the first time, that file does not exist, and it is handled in a node in the pipeline. Kedro throws an missing file error here at this time. Is there a way this can be handled through Kedro? Maybe add an catalog parameter missing=True or optional=True, and Kedro can safely ignore the file? How I currently implemented the solution was to create an empty...

Okay SO gone crazy again

yep, deleting

how can i find kedro_cli.py in my project? i installed kedro using pip install and i made a new project using (kedro new). i was able to run (kedro run --pipeline myPipelineName) from terminal in pycharm or from cmd when i make cd to the project. but i need to run it from my php file using shell_exec(). i need kedro_cli.py but i can't find it. i don't know what to do. tried running C:\\xampp\\htdocs\\webmap\\python\\detect\\venv\\Scripts\\python.py C:\xampp\htdocs\webmap\python\detect\src \detect\__main__.py run --pipeline...

how do kedro catalog properties get passed to Datasets when pipeline is executed I'm trying to create two custom kedro DataSets and I'm not quite sure how to configure the init methods. The first DataSet is simple, its a readonly, fetch rss items from a static URL. catalog entry: rss_feed_extract: type: kedro_workbench.extras.datasets.RSSDataSet.RSSFeedExtract url: https://api.msrc.microsoft.com/update-guide/rss class definition class RSSFeedExtract(AbstractDataSet): def __init__(self, url: str): self._url = url Dict[str, Any]: raw_rss_feed =...

how to create kedro catalog entry + custom DataSet init method with values from credentials.yml I'm trying to create a custom DataSet class within the kedro framework. I need some help understanding how to combine values from the credentials.yml file. what is the kedro way of handling the 'mongo_url' property in the catalog entry? how do i map the values from credentials to the catalog entry? what does the class init method look like? catalog.yml rss_feed_load: type: kedro_workbench.extras.datasets.RSSDataSet.RSSFeedLoad mongo_url:...

logging in python and kedro, how to log only DEBUG info to a file and INFO to console I'm trying to configure logging so that INFO level messages go to the console and DEBUG level messages go to a file instead. So far, I am able to get working INFO to console and DEBUG to file, the problem is that the DEBUG is also being output to the console and I'm not sure why. In particular, I'm using kedro to organize my project and it has some other features I'm trying to figure out. The following works insofar as the console gets INFO and only DEBUG level messages are saved in the file....

Getting Kedro globals.yml to work with OmegaConfigLoader I just started evaluating Kedro for use, and I began with a small project where I read data from a MS-SQL Server. The pipeline will run with a few months in between, and with new date ranges every time. To get this parameter (date) into the pipeline I looked into using globals.yml. The kedro run command works, I have created just one node that loads the data. I load some parameters from globals.yml to use in catalog.yml. If I use the standard lines in settings.py from kedro.config import...

in Kedro, how to handle tar.gz archives from the web I have a tar.gz file that I am downloading from this link: http://ocelma.net/MusicRecommendationDataset/lastfm-1K.html What is the best way to fully integrate this TSV data into kedro, perhaps with an API dataset first, and then a node to extract it? Tar.gz files are not a default supported kedro dataset type.

Define column names when reading a spark dataset in kedro With kedro, how can I define the column names when reading a spark.SparkDataSet? below my catalog.yaml. user-playlists: type: spark.SparkDataSet file_format: csv filepath: data/01_raw/lastfm-dataset-1K/userid-timestamp-artid-artname-traid-traname.tsv load_args: sep: "\t" header: False # schema: # filepath: conf/base/playlists-schema.json save_args: index: False I have been trying to use the following schema, but it doesn't seem to be accepted (schema...

in kedro / pyspark how to use MemoryDataset I am trying to use a MemoryDataset with kedro, in order to not save the intermeiate result to disk. # nodes.py def preprocess_format_tracksessions(tracksess: DataFrame, userid_profiles:pd.DataFrame , parameters: Dict) -> MemoryDataset: In the pipeline I am defining the node output and inputs: # pipeline.py def create_pipeline(**kwargs) -> Pipeline: return pipeline([ node( func=preprocess_format_tracksessions, inputs= ["track_sessions",...

How to import Custom json encoder class into data cataloge I’ve a df which stores lists in a column. I am saving the df with all columns in json using config_new:   type: json.JSONDataSet   filepath: data/01_raw/new_config.json   save_args:     indent: 6   It’s saving all columns ok , except the column with list as string. As in: “T”:[{ “Col1”: “9” “Col2”: “[“7”,”9”,”0”,”5”]” }] As you can see above col2 list is coming out as string I am using json encoder class as below in a python script and saving it under src : import json  ...

Kedro simple project run error: ModuleNotFoundError followed by ValueError I am trying to setup a simple python ML project in Kedro. It is very simple, one pipeline of three nodes, a data loading node, a model computing node and a model evaluating node that just prints the accuracy. This is just to learn to use Kedro for an upcoming project. Despite the simplicity I am struggling to make this work. Basically my kedro run outputs ModuleNotFoundError: No module named 'kedro_mnist.pipelines.' followed by KeyError: 'pipeline' The above exception was the direct cause...

Kedro viz blank page I created a sample pipeline that works correctly with kedro run, however when I try to visualise it with kedro viz I'm getting basically a blank page, even though the terminal doesn't show a single error. The only detail I've found was in the inspection mode: The operating system I'm running it on is windows 10, when I launch it on WSL everything is completely normal. Any ideas why it is happening and how to solve it?

KEDRO - How to specify an arbitrary binary file in catalog.yml? I'm currently working on a datascience project using LLMs (Large language models). Weights for models usually come in different formats, most frequently .bin or .gguf, and I'd like to keep it that way. However the only way to store binary files I know is to use type: pickle.PickleDataset like so test_model: # simple example without compression type: pickle.PickleDataSet filepath: data/07_model_output/test_model.pkl backend: pickle I'm not okay with that as I want my model files to be...