본문 바로가기

Data Engineering

 (45)
[Docker] Dockerfile, docker build, Ruby [rocky@localhost ~]$ cd docker_img[rocky@localhost docker_img]$ vi Dockerfile [rocky@localhost docker_img]$ docker build .[+] Building 2.3s (5/5) FINISHED docker:default => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 76B ..
[Docker] Docker Compose [root@localhost ~]# curl -L "https://github.com/docker/compose/releases/download/v2.27.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0100..
[Docker] Docker Image & localhost 접속 Wordpress[root@localhost ~]# docker stop mysqlmysql[root@localhost ~]# docker rm mysqlmysql[root@localhost ~]# docker run -d -p 3307:3306 -e MYSQL_ALLOW_EMPTY_PASSWORD=password --name mysql mysql:5.740abd55b52329af3b4542db4a7c445231f80412ba868bd0d64cdbb56a76a5aad[root@localhost ~]# mysql -h127.0.0.1 -uroot -pmysql> create database wp CHARACTER SET utf8;mysql> SHOW VARIABLES LIKE 'validate_passwo..
[Docker] Docker 설치 및 환경설정 sudo dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.reposudo dnf install docker-ce docker-ce-cli containerd.io[root@localhost rocky]# usermod -aG docker rocky설치된 docker version 확인systemctl start dockersystemctl enable dockernewgrp dockerdocker run hello-world[root@localhost ~]# docker run --rm -it ubuntu:16.04 /bin/bashUnable to find image 'ubuntu:16.04' locally..
[Kafka] Kafka에서 생성된 데이터를 일정한 시간 단위로 mysql 테이블에 저장하기 Topic 1 from kafka import KafkaProducer import numpy as np from time import time,sleep import os import json producer = KafkaProducer(bootstrap_servers = 'localhost:9092', value_serializer = lambda v: json.dumps(v).encode('utf-8')) count = 0 while True: producer.send('topic1',value=np.random.normal()) sleep(.5) count += 1 if count % 10 ==0: print("topic1 producer......") # for data in temp: # pr..
[Kafka] json 데이터 처리하기 1. json 파일 준비 data.json { "p1": [ {"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}, {"id": 3, "name": "Charlie"} ], "p2": [ {"id": 4, "name": "David"}, {"id": 5, "name": "Emma"}, {"id": 6, "name": "Frank"} ], "p3": [ {"id": 7, "name": "Grace"}, {"id": 8, "name": "Henry"}, {"id": 9, "name": "Ivy"} ] } 이 json 데이터를 Producer가 토픽으로 전달하고 Consumer를 통해 받을 것이다. 2. Topic 생성 # KafkaAdminClient : 클러스터와 ..
[Kafka] mysql과 연동하여 Producer, Consumer 실행 1. 사용할 module install !pip install kafka-python !pip install pymysql 2. Topic 생성 from kafka import KafkaProducer, KafkaConsumer from kafka.admin import KafkaAdminClient, NewTopic import json import pymysql # create topic def create_topics(topics, bootstrap_servers = 'localhost:9092'): admin_client = KafkaAdminClient(bootstrap_servers=bootstrap_servers) # 새로운 토픽 리스트 생성 topic_list = [ NewTopic(top..
[Kafka] python으로 Kafka 실행하기 1. Producer 생성 Kafka 라이브러리 설치 # 카프카 라이브러리 설치 !pip install kafka-python Topic이 읽을 파일 생성 1부터 1000까지 숫자들이 들어있는 파일을 생성한다. # 토픽이 읽을 파일을 생성 with open('new-topic.txt', 'w') as f: for i in range(1,1001): f.write(str(i) + '\n') Producer 생성 # 프로듀서 생성(토픽으로 데이터를 전송) from kafka import KafkaProducer import json import time from csv import reader class MessageProducer: def __init__(self,broker,topic): self.bro..

티스토리툴바