BH 1.5.1 rytec EPG issues(tar incorrect)

B

bh4me2

Vu+ Newbie
Any explanation for this, retrieved from rytec to a holding server:
08-05-2011 04:34 11.753.715 epg.dat
08-05-2011 06:35 4.143.166 epg.dat.gz

Then when BH picks it up from the holding server:
08-05-2011 13:05 3.330.701 epg.dat
08-05-2011 13:05 4.143.166 epg.dat.gz

The dat is alot smaller.
 
Matrix10

Matrix10

Administrator
Our BH EPG works differently from CrossEPG and uses direct file
(already converted) is the fastest EPG system for VU duo.
 
Matrix10

Matrix10

Administrator
File Packaging we use to reduce the file size so that we could speed up download
and reduce server load.
 
B

bh4me2

Vu+ Newbie
You are talking about something else, the epg.dat is not correctly unpacked by BH, today the same thing, a lot of epg is missing, on the holding server 11.703.797 epg.dat and on the box 2.517.637 epg.dat both from the same gz file. Is there another way to place this file unpacked on the box and tell BH to process it?
 
Matrix10

Matrix10

Administrator
bh4me2 said:
You are talking about something else, the epg.dat is not correctly unpacked by BH, today the same thing, a lot of epg is missing, on the holding server 11.703.797 epg.dat and on the box 2.517.637 epg.dat both from the same gz file. Is there another way to place this file unpacked on the box and tell BH to process it?
Click to expand...

Hmmm

To understand we discuss BH EPG and not CrossEPG.
If I understand, you have download the same file from your PC and with VU duo
(I have not test this)
and then you get different sizes ???
But this size you have to look after E2 restart and not direct after download.

I think you have the other problem because this I have not seen so far.
If the server file is OK will be unpacked without problems inside image
provided that you have a place on USB and that the file is correctly dowloded.

The same or a similar method (download and unpack) is used for all other downloads
including plugins, skins etc.
So something else is wrong or I have overlooked something.
But Im not sure Im not coder for image EPG
You can try to use Epg backup for manual file manual installation
If you have good file from Rytec server
(and I have not tested this) Im not sure if this will work.
This file you have to unpack on your PC and rename to epg.dat.bak
FTP this file to /media/USB for EPG on USB
and try to restore with Black Hole Epg Backup.
 
B

bh4me2

Vu+ Newbie
I'll try the restore, but it would be nice if I can trigger epg.dat processing by doing 'something' after manually placing the correct epg.dat.

But this is whats going on:

First I had setup BH to pickup epg.dat.gz from rytec, works ok for 2 or 3 days then the box resets with a crash.

So I decided to use a holding server to see whats going on like this:

del epg.dat.gz
del epg.dat
wget www.xmltvepg.be/benluk/epg.dat.gz
if errorlevel 1 echo Wget_error
if errorlevel 1 goto end3
pkzipc -ext -nofix epg.dat.gz epg.dat
if errorlevel 1 echo Zip_error
if errorlevel 1 goto end3
copy /b/y epg.dat.gz C:\pathtolocalwebserver\epg
del /f/q \\bm750\Harddisk\epg.dat
del /f/q \\bm750\Harddisk\epg.dat.gz
:end3

At this point my local webserver holds a valid and CRC tested epg.dat.gz which gets picked up by the BH box a few hours later.

After BH picks up the gz file and processes it, something is going wrong when BH unzips the gz file, hence the filesize differences.
 
B

bh4me2

Vu+ Newbie
Ea. what I mean by manual update is something like this:

pkzipc -ext -nofix epg.dat.gz epg.dat
copy /b/y epg.dat \\bm750\Harddisk\epg.dat
[do something / send command to vu+ to tell it a new epg.dat file is there]

Assuming here that this problem is a BH unzip issue. ea. maybe there is not enough ram to process the unzip command.
 
Matrix10

Matrix10

Administrator
These are already known problems and is already written on forum.
Enigma uses a protection mechanism if you insert many
EPG data in to memory.Therefore, there is a enigma crash .
You can try to use the SWAP ,some say it helps.
I do not have positive results of this test.
I think that If you use more than say + - 10MB EPG
(+ -3 providers) use CrossEPG
I have never enigma crash for this reason because I do not use
Many providers.

For the rest Im sorry I do not have time to test.

P.s
First I had setup BH to pickup epg.dat.gz from rytec, works ok for 2 or 3 days then the box resets with a crash.
Click to expand...

You can use EPG backup for this,and manual restore .
Our coder did not want to do this automatically
I think because we will have even bigger problem
and enter the crash circle .
 
B

bh4me2

Vu+ Newbie
Ok. Well I tried the restore which did work initially but just before it finishes the box crashes without a crashlog but there are entries in \Configuration\log\messages:

May 10 17:37:52 bm750 user.warn kernel: oom-killer: gfp_mask=0x201d2, order=0
May 10 17:37:52 bm750 user.warn kernel: Call Trace:
May 10 17:37:52 bm750 user.warn kernel: [<80011390>] dump_stack+0x20/0x50
May 10 17:37:52 bm750 user.warn kernel: [<8007a5fc>] out_of_memory+0x2a8/0x2ec
May 10 17:37:52 bm750 user.warn kernel: [<8007cd90>] __alloc_pages+0x360/0x374
May 10 17:37:52 bm750 user.warn kernel: [<8007ef80>] __do_page_cache_readahead+0x1b4/0x360
May 10 17:37:52 bm750 user.warn kernel: [<80076474>] filemap_nopage+0x354/0x714
May 10 17:37:52 bm750 user.warn kernel: [<8008a0cc>] __handle_mm_fault+0x15c/0xcb8
May 10 17:37:52 bm750 user.warn kernel: [<8001584c>] do_page_fault+0x12c/0x3b0
May 10 17:37:52 bm750 user.warn kernel: [<8001682c>] tlb_do_page_fault_0+0x10c/0x114
May 10 17:37:52 bm750 user.warn kernel: Mem-info:
May 10 17:37:52 bm750 user.warn kernel: DMA per-cpu:
May 10 17:37:52 bm750 user.warn kernel: cpu 0 hot: high 90, batch 15 used:17
May 10 17:37:52 bm750 user.warn kernel: cpu 0 cold: high 30, batch 7 used:24
May 10 17:37:52 bm750 user.warn kernel: cpu 1 hot: high 90, batch 15 used:48
May 10 17:37:52 bm750 user.warn kernel: cpu 1 cold: high 30, batch 7 used:0
May 10 17:37:52 bm750 user.warn kernel: DMA32 per-cpu: empty
May 10 17:37:52 bm750 user.warn kernel: Normal per-cpu: empty
May 10 17:37:52 bm750 user.warn kernel: HighMem per-cpu: empty
May 10 17:37:52 bm750 user.warn kernel: Free pages: 2024kB (0kB HighMem)
May 10 17:37:52 bm750 user.warn kernel: Active:27905 inactive:573 dirty:0 writeback:0 unstable:0 free:506 slab:3271 mapped:9 pagetables:134
May 10 17:37:52 bm750 user.warn kernel: DMA free:2024kB min:2048kB low:2560kB high:3072kB active:112132kB inactive:2036kB present:262144kB pages_scanned:21116 all_unreclaimable? no
May 10 17:37:52 bm750 user.warn kernel: lowmem_reserve[]: 0 0 0 0
May 10 17:37:52 bm750 user.warn kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 10 17:37:52 bm750 user.warn kernel: lowmem_reserve[]: 0 0 0 0
May 10 17:37:52 bm750 user.warn kernel: Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 10 17:37:52 bm750 user.warn kernel: lowmem_reserve[]: 0 0 0 0
May 10 17:37:52 bm750 user.warn kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 10 17:37:52 bm750 user.warn kernel: lowmem_reserve[]: 0 0 0 0
May 10 17:37:52 bm750 user.warn kernel: DMA: 45*4kB 8*8kB 1*16kB 1*32kB 3*64kB 2*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2276kB
May 10 17:37:52 bm750 user.warn kernel: DMA32: empty
May 10 17:37:52 bm750 user.warn kernel: Normal: empty
May 10 17:37:52 bm750 user.warn kernel: HighMem: empty
May 10 17:37:52 bm750 user.warn kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
May 10 17:37:52 bm750 user.warn kernel: Free swap = 0kB
May 10 17:37:52 bm750 user.warn kernel: Total swap = 0kB
May 10 17:37:52 bm750 user.warn kernel: Free swap: 0kB
May 10 17:37:52 bm750 user.warn kernel: 65536 pages of RAM
May 10 17:37:52 bm750 user.warn kernel: 0 pages of HIGHMEM
May 10 17:37:52 bm750 user.warn kernel: 30166 reserved pages
May 10 17:37:52 bm750 user.warn kernel: 201 pages shared
May 10 17:37:52 bm750 user.warn kernel: 0 pages swap cached
May 10 17:37:52 bm750 user.err kernel: Out of Memory: Kill process 670 (enigma2.sh) score 1323 and children.
May 10 17:37:52 bm750 user.err kernel: Out of memory: Killed process 673 (enigma2).

So a manual restore is not the answer, it is a bit moot to restrict epg to 10mb.... we watch channels from 3 sats and use the epg as well. But if a swapfile can work can you explain how to set that up ?
 
Matrix10

Matrix10

Administrator
You can test if you want .

SWAP file you can make as follows

blue button>>blue button>>Swap File Manger>>Green Create

RED activate and yellow autostart
 
Armageddon

Armageddon

BH Lover
bh4me2 said:
Ok. Well I tried the restore which did work initially but just before it finishes the box crashes without a crashlog but there are entries in \Configuration\log\messages:

May 10 17:37:52 bm750 user.warn kernel: oom-killer: gfp_mask=0x201d2, order=0
May 10 17:37:52 bm750 user.warn kernel: Call Trace:
May 10 17:37:52 bm750 user.warn kernel: [<80011390>] dump_stack+0x20/0x50
May 10 17:37:52 bm750 user.warn kernel: [<8007a5fc>] out_of_memory+0x2a8/0x2ec
May 10 17:37:52 bm750 user.warn kernel: [<8007cd90>] __alloc_pages+0x360/0x374
May 10 17:37:52 bm750 user.warn kernel: [<8007ef80>] __do_page_cache_readahead+0x1b4/0x360
May 10 17:37:52 bm750 user.warn kernel: [<80076474>] filemap_nopage+0x354/0x714
May 10 17:37:52 bm750 user.warn kernel: [<8008a0cc>] __handle_mm_fault+0x15c/0xcb8
May 10 17:37:52 bm750 user.warn kernel: [<8001584c>] do_page_fault+0x12c/0x3b0
May 10 17:37:52 bm750 user.warn kernel: [<8001682c>] tlb_do_page_fault_0+0x10c/0x114
May 10 17:37:52 bm750 user.warn kernel: Mem-info:
May 10 17:37:52 bm750 user.warn kernel: DMA per-cpu:
May 10 17:37:52 bm750 user.warn kernel: cpu 0 hot: high 90, batch 15 used:17
May 10 17:37:52 bm750 user.warn kernel: cpu 0 cold: high 30, batch 7 used:24
May 10 17:37:52 bm750 user.warn kernel: cpu 1 hot: high 90, batch 15 used:48
May 10 17:37:52 bm750 user.warn kernel: cpu 1 cold: high 30, batch 7 used:0
May 10 17:37:52 bm750 user.warn kernel: DMA32 per-cpu: empty
May 10 17:37:52 bm750 user.warn kernel: Normal per-cpu: empty
May 10 17:37:52 bm750 user.warn kernel: HighMem per-cpu: empty
May 10 17:37:52 bm750 user.warn kernel: Free pages: 2024kB (0kB HighMem)
May 10 17:37:52 bm750 user.warn kernel: Active:27905 inactive:573 dirty:0 writeback:0 unstable:0 free:506 slab:3271 mapped:9 pagetables:134
May 10 17:37:52 bm750 user.warn kernel: DMA free:2024kB min:2048kB low:2560kB high:3072kB active:112132kB inactive:2036kB present:262144kB pages_scanned:21116 all_unreclaimable? no
May 10 17:37:52 bm750 user.warn kernel: lowmem_reserve[]: 0 0 0 0
May 10 17:37:52 bm750 user.warn kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 10 17:37:52 bm750 user.warn kernel: lowmem_reserve[]: 0 0 0 0
May 10 17:37:52 bm750 user.warn kernel: Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 10 17:37:52 bm750 user.warn kernel: lowmem_reserve[]: 0 0 0 0
May 10 17:37:52 bm750 user.warn kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 10 17:37:52 bm750 user.warn kernel: lowmem_reserve[]: 0 0 0 0
May 10 17:37:52 bm750 user.warn kernel: DMA: 45*4kB 8*8kB 1*16kB 1*32kB 3*64kB 2*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2276kB
May 10 17:37:52 bm750 user.warn kernel: DMA32: empty
May 10 17:37:52 bm750 user.warn kernel: Normal: empty
May 10 17:37:52 bm750 user.warn kernel: HighMem: empty
May 10 17:37:52 bm750 user.warn kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
May 10 17:37:52 bm750 user.warn kernel: Free swap = 0kB
May 10 17:37:52 bm750 user.warn kernel: Total swap = 0kB
May 10 17:37:52 bm750 user.warn kernel: Free swap: 0kB
May 10 17:37:52 bm750 user.warn kernel: 65536 pages of RAM
May 10 17:37:52 bm750 user.warn kernel: 0 pages of HIGHMEM
May 10 17:37:52 bm750 user.warn kernel: 30166 reserved pages
May 10 17:37:52 bm750 user.warn kernel: 201 pages shared
May 10 17:37:52 bm750 user.warn kernel: 0 pages swap cached
May 10 17:37:52 bm750 user.err kernel: Out of Memory: Kill process 670 (enigma2.sh) score 1323 and children.
May 10 17:37:52 bm750 user.err kernel: Out of memory: Killed process 673 (enigma2).

So a manual restore is not the answer, it is a bit moot to restrict epg to 10mb.... we watch channels from 3 sats and use the epg as well. But if a swapfile can work can you explain how to set that up ?
Click to expand...

I hava also problems with the 10Mb limit, i´ve tried swapfil, but both my boxes (Solo + Duo) slows down when using a swapfile. Swapsize builds up and the box can get really slow after a few days, most frequently on Solo. Sometimes i need to hard reset as I´m not able to navigate in the menus. I´m using 2 Rytec providers. I´ve had the same problem in all images since 1.36 until 1.5.1. I can not say anything about CrossEGP yet. Haven´t used it enough to tell yet.
 
B

bh4me2

Vu+ Newbie
Well I just tried a few things and it all crashes the box.... so I tried crossepg again (last time I tried crossepg the box also crashed) but this time crossepg is working, no crashes and its loading about 30mb of epg data, so far so good.
 
Matrix10

Matrix10

Administrator
I am not using the swap so I do not know
I never used

I'm sorry but I can not help you
Try CrossEPG Perhaps you can use multiple EPG providers.
 
B

bh4me2

Vu+ Newbie
And again after a few days crossepg is crashing the box again... sigh... but I've found a workaround that might be a bit more stable (bypass download, unzip and some parsing), I'm using my rytec holding server again and only copy over the unpacked epg.dat file, then inside crossepg I use the "force epg.dat conversion now", done in 10 seconds and no crash so far. (it is actually a import so you can repeat this every day) 217000 events 20mb database.

Question: can I call that option inside crossepg (force epg.dat conversion now) via bhcron and if yes what do I put into it?
 
You must log in or register to reply here.
Top