tokenizer, streams, bugfixes WAS: FW: patch

[Lendvai Attila]

ok, here i forward stuff to the list and will do so later. i just tought it's not really useful until you approve it and put it in cvs...
 
note: withOpenNamed sounds quite strange for me. openNamed is much more natural, but it breaks the with* conventions...
 
- 101
 
ps: sorry for the html mail 

	Lendvai Attila <Attila.Lendvai at netvisor.hu> said:
	
	> hi!
	> 
	> i've got a patch with some changes, apply what is useful.
	> 
	> i had to make some of these to make stream >> bag writer work, but i'm not
	sure this is the right way.
	
	Thanks! Please include the mailing list when you send patches in the future.

	
	I should mention that Attila has been testing out some utilities like
	src/tokenizer.slate, particularly figuring out how to take easy word-counts of
	file contents. Here's an illustration:
	
	File withOpenNamed: 'license.txt' do: [| :f results |
	    results: Bag newEmpty writer. "Holds elements and counts."
	    (f reader splitWith: {$\s. $\t. $\n}) >> results.
	    results contents].
	
	I realize from this that #>> should be returning the target stream, so that
	you can just immediately grab the results without needing another local slot
	to refer to it.
	
	#splitWith: is an invocation of the tokenizer to get a stream-like analogue of
	Sequence splitWith:. Like other Stream/Collection methods (collect:, select:,
	reject:), it follows a rule that calls to collections return collections, and
	streams for streams.
	
	Also, the #split method is an easier form of #splitWith: using default
	whitespace for separators.
	
	So, once the #>> change is in, the script should be in its shortest form as:
	
	File withOpenNamed: 'license.txt' do: [| :f |
	    (f reader split >> Bag newEmpty writer) contents].
	
	
	

-------------- next part --------------
Index: bootstrap/bag.slate
===================================================================
RCS file: /var/lib/cvs/slate/slate/bootstrap/bag.slate,v
retrieving revision 1.2
diff -c -r1.2 bag.slate
*** bootstrap/bag.slate	18 Jul 2004 00:45:06 -0000	1.2
--- bootstrap/bag.slate	19 Jul 2004 21:09:54 -0000
***************
*** 127,133 ****
  [| counts |
    counts: (SortedArray newSizeOf: b contents).
    counts sortBlock: [| :x :y | x >= y].
!   c contents keysAndValuesDo: [| :key :val |
      counts add: (val -> key)].
    counts
  ].
--- 127,133 ----
  [| counts |
    counts: (SortedArray newSizeOf: b contents).
    counts sortBlock: [| :x :y | x >= y].
!   b contents keysAndValuesDo: [| :key :val |
      counts add: (val -> key)].
    counts
  ].
Index: bootstrap/iterator.slate
===================================================================
RCS file: /var/lib/cvs/slate/slate/bootstrap/iterator.slate,v
retrieving revision 1.3
diff -c -r1.3 iterator.slate
*** bootstrap/iterator.slate	18 Jul 2004 06:43:45 -0000	1.3
--- bootstrap/iterator.slate	19 Jul 2004 21:09:54 -0000
***************
*** 41,46 ****
--- 41,53 ----
  ExtensibleCollection WriteStream addSlot: #collection.
  "The target of the WriteStream."
  
+ ws@(ExtensibleCollection WriteStream traits) on: c
+ "Targets the collection."
+ [
+   ws collection: c.
+   ws
+ ].
+ 
  ws@(ExtensibleCollection WriteStream traits) nextPut: obj
  [ws collection add: obj].
  
Index: bootstrap/stream.slate
===================================================================
RCS file: /var/lib/cvs/slate/slate/bootstrap/stream.slate,v
retrieving revision 1.8
diff -c -r1.8 stream.slate
*** bootstrap/stream.slate	18 Jul 2004 06:23:59 -0000	1.8
--- bootstrap/stream.slate	19 Jul 2004 21:09:54 -0000
***************
*** 4,12 ****
  "The shared protocol of all Stream objects, also providing a common
  instantiation protocol."
  
! s@(Stream traits) newOn: _
  "Create a new stream of the same kind targetted on the given object."
! [overrideThis].
  
  s@(Stream traits) on: _
  "(Re-)target the stream to some object. This modifies it in-place.
--- 4,16 ----
  "The shared protocol of all Stream objects, also providing a common
  instantiation protocol."
  
! s@(Stream traits) newOn: c
  "Create a new stream of the same kind targetted on the given object."
! [| new |
!   new: s clone.
!   new on: c.
!   new
! ].
  
  s@(Stream traits) on: _
  "(Re-)target the stream to some object. This modifies it in-place.
Index: src/dictionary.test
===================================================================
RCS file: /var/lib/cvs/slate/slate/src/dictionary.test,v
retrieving revision 1.4
diff -c -r1.4 dictionary.test
*** src/dictionary.test	7 Mar 2004 04:14:44 -0000	1.4
--- src/dictionary.test	19 Jul 2004 21:09:54 -0000
***************
*** 65,71 ****
    t assert: (d1 at: 'January') = (d2 at: 'January') description: 'Value at January differs'.
    t assert: (d1 at: 'December') = (d2 at: 'December') description: 'Value at December differs'.
  
!   (d1 at: 'January' ifAbsent: [t signalFailureDescription: 'at:ifAbsent: block should not get evaluated here']) = 31.
  
    t assert: (d1 keyAtValue: 29) = (d2 keyAtValue: 29) description: 'keyAtValue: 29 differs'.
  
--- 65,71 ----
    t assert: (d1 at: 'January') = (d2 at: 'January') description: 'Value at January differs'.
    t assert: (d1 at: 'December') = (d2 at: 'December') description: 'Value at December differs'.
  
!   t assert: (d1 at: 'January' ifAbsent: [t signalFailureDescription: 'at:ifAbsent: block should not get evaluated here']) = 31 description: 'Unexpected value at January'.
  
    t assert: (d1 keyAtValue: 29) = (d2 keyAtValue: 29) description: 'keyAtValue: 29 differs'.
  
Index: src/test.slate
===================================================================
RCS file: /var/lib/cvs/slate/slate/src/test.slate,v
retrieving revision 1.26
diff -c -r1.26 test.slate
*** src/test.slate	7 Mar 2004 03:28:58 -0000	1.26
--- src/test.slate	19 Jul 2004 21:09:54 -0000
***************
*** 73,81 ****
  tc@(TestCase traits) executeShould: block inScopeOf: cond
  "Answers whether executing the block raises the given condition."
  [
!   [block on: cond do: [| :c | ^ True]]
!     on: Error do: [| :c | ^ False].
!   "TODO: could use breakOn:"
    False
  ].
  
--- 73,80 ----
  tc@(TestCase traits) executeShould: block inScopeOf: cond
  "Answers whether executing the block raises the given condition."
  [
!   block on: cond do: [| :c | ^ True].
! 
    False
  ].