pythonでinca(dat)データの読み取り

はじめに

仕事でdatの拡張子のデータを扱うことが多い。これを開くにはやや面倒なソフト(mda)を立ち上げる必要があり、何かできないかと思っていた。するとmdfreaderというものを使えば読めると聞き、色々調べてみた。mdfとは「Measurement Data Format」の略みたいで測定データを指しているようだ。datも測定データだし、もしかしたら行けるのではと思いやってみた。ちなみにfamosで扱うdatとは違います。

こちらをみるとちょっと期待が持てた
etasHP

とりあえずコード

こちらです。
参考にしたサイトはこちら。

【完全版】pythonでMDFファイルのCANデータをcsv化する
これ通りやってもうまくいかなかった。のでこのページが生まれた。

Git
Gitの↓のほうにいくと色々なコマンド紹介がある。これを参考にしてどうにかなった。

動作未確認。。。

import pandas as pd
import mdfreader

fname=input("input fname:")
print(fname)

mode=input("1:make list, 2:list to csv, 3:all csv")

if mode==1:
    # 信号リスト作成
    yop=mdfreader.MdfInfo()
    df_sig=pd.DataFrame(yop.list_channels(fname+".dat"))
    df_sig.to_csv("temp.csv")

if mode==2:
    # 信号リストの書式変換 
    list_data=pd.read_csv("temp.csv").values.tolist()
    list_data=[flatten for inner in list_data for flatten in inner]

    # MDF読み込み
    inst=mdfreader.Mdf(fname+".dat", channel_list=list_data,
                   convert_after_read=False, metadata=2)

if mode==3:
    # 全ての場合
    inst=mdfreader.Mdf(fname+".dat")

if mode==2 or 3:
    #同名のファイル作成。1行置きに空白。。    
    inst.export_to_csv(sampling=0.01)
    #1行置きが消える
    df_data=pd.read_csv(fname+".csv",encoding='CPC932')
    df_data.to_csv(fname+".csv")

参考にgitの抜粋をちょっと載せておく。

    import mdfreader
    # loads whole mdf file content in yop mdf object.
    yop=mdfreader.Mdf('NameOfFile')
    # you can print file content in ipython with a simple:
    yop
    # alternatively, for max speed and smaller memory footprint, read only few channels
    yop=mdfreader.Mdf('NameOfFile', channel_list=['channel1', 'channel2'], convert_after_read=False)
    # also possible to keep data compressed for small memory footprint, using Blosc module
    yop=mdfreader.Mdf('NameOfFile', compression=True)
    # for interactive file exploration, possible to read the file but not its data to save memory
    yop=mdfreader.Mdf('NameOfFile', no_data_loading=True) # channel data will be loaded from file if needed
    # parsing xml metadata from mdf4.x for many channels can take more than just reading data.
    # You can reduce to minimum metadata reading with below argument (no source information, attachment, etc.) 
    yop=mdfreader.Mdf('NameOfFile', metadata=0)  # 0: full, 2: minimal
    # only for mdf4.x, you can search for the mdf key of a channel name that can have been recorded by different sources
    yop.get_channel_name4('channelName', 'source path or name')  # returns list of mdf keys
    # to yield one channel and keep its content in mdf object
    yop.get_channel('channelName')
    # to yield one channel numpy array
    yop.get_channel_data('channelName')
    # to get file mdf version
    yop.MDFVersionNumber
    # to get file structure or attachments, you can create a mdfinfo instance
    info=mdfreader.MdfInfo()
    info.list_channels('NameOfFile') # returns only the list of channels
    info.read_info('NameOfFile') # complete file structure object
    yop.info # same class is stored in mdfreader class
    # to list channels names after reading
    yop.keys()
    # to list channels names grouped by raster, below dict mdf attribute contains
    # pairs (key=masterChannelName : value=listOfChannelNamesForThisMaster)
    yop.masterChannelList
    # quick plot or subplot (with lists) of channel(s)
    yop.plot(['channel1',['channel2','channel3']])
    # file manipulations
    yop.resample(0.1)
    # or
    yop.resample(master_channel='master3')
    # keep only data between begin and end
    yop.cut(begin=10, end=15)
    # export to other file formats :
    yop.export_to_csv(sampling=0.01)
    yop.export_to_NetCDF()
    yop.export_to_hdf5()
    yop.export_to_matlab()
    yop.export_to_xlsx()
    yop.export_to_parquet()
    # return pandas dataframe from master channel name
    yop.return_pandas_dataframe('master_channel_name')
    # converts data groups into pandas dataframes and keeps it in mdf object
    yop.convert_to_pandas()
    # drops all the channels except the one in argument
    yop.keep_channels({'channel1','channel2','channel3'})
    # merge 2 files
    yop2=mdfreader.Mdf('NameOfFile_2')
    yop.merge_mdf(yop2)
    # can write mdf file after modifications or creation from scratch
    # write4 and write3 also allow to convert file versions
    yop.write('NewNameOfFile')  # write in same version as original file after modifications
    yop.write4('NameOfFile', compression=True)  # write mdf version 4.1 file, data compressed
    yop.write3()  # write mdf version 3 file
    yop.attachments  # to get attachments, embedded or paths to files 

おわりに

python系はアクセス数が伸びるのでこちらもちょい期待。