[project2_ksa_tss-b4b4] Details ============================ Generated On: 2024-10-22 02:42:48 UTC TML Solution DAG Parameters' Details: User Chosen Parametets ---------------------------- STEP 1: Get TML Core Params: `tml_system_step_1_getparams_dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - solutionname - project2_ksa_tss-b4b4 * - solutiontitle - My Solution Title * - solutiondescription - This is an awesome real-time solution built by TSS * - brokerhost - 127.0.0.1 * - brokerport - 9092 * - cloudusername - None * - ingestdatamethod - LOCALFILE STEP 2: Create Kafka Topics: `tml_system_step_2_kafka_createtopic_dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - companyname - Otics * - myname - Sebastian * - myemail - Sebastian.Maurice * - mylocation - Toronto * - replication - 1 * - numpartitions - 1 * - enabletls - 1 * - microserviceid - * - raw_data_topic - iot-raw-data * - preprocess_data_topic - iot-preprocess,iot-preprocess2 * - ml_data_topic - ml-data * - prediction_data_topic - prediction-data STEP 3: `Produce to Kafka Topics `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - PRODUCETYPE - LOCALFILE * - TOPIC - iot-raw-data * - PORT - 41391 * - IDENTIFIER - TML solution,/rawdata/IoTData.txt * - HTTPADDR - https:// * - FROMHOST - ('tml-kas', '127.0.1.1') * - TOHOST - 127.0.1.1 * - CLIENTPORT - Not Applicable * - TSS_CLIENTPORT - Not Applicable * - TML_CLIENTPORT - Not Applicable STEP 4: Preprocesing Data: `tml-system-step-4-kafka-preprocess-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - raw_data_topic - iot-raw-data * - preprocess_data_topic - iot-preprocess,iot-preprocess2 * - preprocessconditions - * - delay - 70 * - array - 0 * - saveasarray - 1 * - topicid - -999 * - rawdataoutput - 1 * - asynctimeout - 120 * - timedelay - 0 * - preprocesstypes - anomprob,trend,avg * - pathtotmlattrs - --pathtotmlattrs-- * - identifier - IoT device performance and failures * - jsoncriteria - uid=metadata.dsn,filter:allrecords~subtopics=metadata.property_name~values=datapoint.value~identifiers=metadata.display_name~datetime=datapoint.updated_at~msgid=datapoint.id~latlong=lat:long STEP 4b: Preprocesing Data: `tml-system-step-4b-kafka-preprocess-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - raw_data_topic - --raw_data_topic2-- * - preprocess_data_topic - --preprocess_data_topic2-- * - preprocessconditions - --preprocessconditions2-- * - delay - --delay2-- * - array - --array2-- * - saveasarray - --saveasarray2-- * - topicid - --topicid2-- * - rawdataoutput - --rawdataoutput2-- * - asynctimeout - --asynctimeout2-- * - timedelay - --timedelay2-- * - preprocesstypes - --preprocesstypes2-- * - pathtotmlattrs - --pathtotmlattrs2-- * - identifier - --identifier2-- * - jsoncriteria - --jsoncriteria2-- STEP 5: Entity Based Machine Learning : `tml-system-step-5-kafka-machine-learning-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - preprocess_data_topic - iot-preprocess,iot-preprocess2 * - ml_data_topic - ml-data * - modelruns - --modelruns-- * - offset - -1 * - islogistic - --islogistic-- * - networktimeout - --networktimeout-- * - modelsearchtuner - --modelsearchtuner-- * - processlogic - --processlogic-- * - dependentvariable - --dependentvariable-- * - independentvariables - --independentvariables-- * - rollbackoffsets - --rollbackoffsets-- * - topicid - -999 * - consumefrom - --consumefrom-- * - fullpathtotrainingdata - --fullpathtotrainingdata-- * - transformtype - --transformtype-- * - sendcoefto - --sendcoefto-- * - coeftoprocess - --coeftoprocess-- * - coefsubtopicnames - --coefsubtopicnames-- STEP 6: Entity Based Predictions: `tml-system-step-6-kafka-predictions-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - preprocess_data_topic - iot-preprocess,iot-preprocess2 * - ml_prediction_topic - --ml_prediction_topic-- * - streamstojoin - --streamstojoin-- * - inputdata - --inputdata-- * - consumefrom - --consumefrom2-- * - offset - -1 * - delay - 70 * - usedeploy - --usedeploy-- * - networktimeout - --networktimeout-- * - maxrows - --maxrows-- * - topicid - -999 * - pathtoalgos - --pathtoalgos-- STEP 7: Real-Time Visualization: `tml-system-step-7-kafka-visualization-dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - vipervizport - 9005 * - topic - iot-preprocess,iot-preprocess2 * - dashboardhtml - dashboard.html * - secure - 1 * - offset - -1 * - append - 0 * - chip - amd64 * - rollbackoffset - 400 STEP 8: `tml_system_step_8_deploy_solution_to_docker_dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - Docker Container - kasunsam/project2_ksa_tss-b4b4-amd64 (https://hub.docker.com/r/kasunsam/project2_ksa_tss-b4b4-amd64) * - Docker Run Command - docker run -d -p 47771:47771 -p 43491:43491 -p 37785:37785 \-\-env TSS=0 \-\-env SOLUTIONNAME=project2_ksa_tss-b4b4 \-\-env SOLUTIONDAG=solution_preprocessing_dag-project2_ksa_tss-b4b4 \-\-env GITUSERNAME=124457235-myseneca \-\-env GITREPOURL=https://github.com/124457235-myseneca/raspberrypi.git \-\-env SOLUTIONEXTERNALPORT=47771 \-\-env CHIP=amd64 \-\-env SOLUTIONAIRFLOWPORT=43491 \-\-env SOLUTIONVIPERVIZPORT=37785 \-\-env DOCKERUSERNAME='kasunsam' \-\-env EXTERNALPORT=41391 \-\-env KAFKACLOUDUSERNAME='' \-\-env VIPERVIZPORT=9005 \-\-env MQTTUSERNAME='ksathugoda.ca@gmail.com' \-\-env AIRFLOWPORT=9000 \-\-env MQTTPASSWORD='' \-\-env KAFKACLOUDPASSWORD='' \-\-env GITPASSWORD='' \-\-env READTHEDOCS='' kasunsam/project2_ksa_tss-b4b4-amd64 STEP 9: `tml_system_step_9_privategpt_qdrant_dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - PrivateGPT Container - --pgptcontainername-- * - PrivateGPT Run Command - --privategptrun-- * - Qdrant Container - --qdrantcontainer-- * - Qdrant Run Command - --qdrantrun-- * - Consumefrom - --consumefrom-- * - pgpt_data_topic - --pgpt_data_topic-- * - offset - -1 * - rollbackoffset - 400 * - topicid - -999 * - enabletls - 1 * - partition - --partition-- * - prompt - --prompt-- * - context - --context-- * - jsonkeytogather - --jsonkeytogather-- * - keyattribute - --keyattribute-- * - keyprocesstype - --keyprocesstype-- * - vectordbcollectionname - --vectordbcollectionname-- * - concurrency - --concurrency-- * - CUDA_VISIBLE_DEVICES - --cuda-- * - pgpthost - --pgpthost-- * - pgptport - --pgptport-- * - hyperbatch - --hyperbatch-- STEP 10: `tml_system_step_10_documentation_dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - Solution Documentation URL - https://project2_ksa_tss-b4b4.readthedocs.io